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ABSTRACT 


This  thesis  conducted  an  examination  related  to  Department  of  Defense  (DOD) 
weapons  systems  production-approval  practices.  Current  practices  result  in  poor  weapons 
systems  production  outcomes  that  reduce  fleet  readiness  in  DOD  weapons  systems 
acquisition.  The  Government  Accountability  Office  (GAO)  has  reported  concerns  related 
to  a  lack  of  manufacturing  knowledge  at  production  start  as  causal  to  poor  production 
outcomes.  A  comparison  of  DOD  practices  against  non-DOD  industrial  production 
approval  processes  addressing  causality  and  improvement  opportunity  provided  new 
insight  not  found  in  acquisition  research.  An  analysis  of  alternatives  identified  best 
practices  to  improve  production  capability  and  readiness.  Key  findings  revealed  that  the 
automotive  production  approval  process  followed  industry  best  practices  that  fully 
addressed  problems  identified  by  the  GAO.  Non-DOD  industries  used  a  more 
prescriptive  Quality  Management  System  (QMS)  that  enabled  a  more  disciplined 
manufacturing  development  and  demonstration  of  production  capability  prior  to 
production  commitment.  Commercial  surveys  in  the  literature  confirmed  the  benefits  of 
the  automotive  prescriptive  QMS.  The  more  successful  QMS  approach  can  be  applied  to 
DOD  acquisition  practices  reducing  costs  and  improving  fleet  readiness. 
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EXECUTIVE  SUMMARY 


Poor  production  outcomes  in  Department  of  Defense  (DOD)  weapon  systems 
acquisition  are  costly  and  adversely  affect  fleet  readiness.  National  security  demands  a 
highly  capable  and  ready  fleet  to  respond  to  a  complex  global  threat  environment.  One 
agency  charged  with  oversight  of  weapon  systems  acquisition  is  the  Government 
Accountability  Office  (GAO).  Annual  GAO  reports  on  select  weapon  programs  show  a 
persistent  and  troubling  observation  revealing  poor  production  outcomes  in  DOD 
acquisition  (Dodaro  2013).  Dodaro  identified  three  reasons  for  DOD  weapons  systems 
poor  production  outcomes:  1)  knowledge  gaps  in  technology,  2)  design  instability  and  3) 
manufacturing  knowledge  gaps.  Only  manufacturing  knowledge  gaps  lacked  DOD 
attention.  Additional  details  reported  by  Dodaro  concerning  manufacturing  causality 
included:  1)  use  of  non-standard  processes,  2)  failure  to  identify  critical  manufacturing 
processes,  and  3)  failure  to  apply  statistical  process  control. 

The  current  state  of  DOD  acquisition  of  weapon  systems  follows  the  defense 
acquisition  system  consisting  of  law,  policy,  instruction  and  other  guidance  documents.  A 
current-state  system  operational  view  is  shown  in  Figure  1.  DOD  guidance  pays  little 
attention  to  manufacturing  during  engineering  and  manufacturing  development  (EMD). 
The  acquisition  process  related  to  manufacturing  development  relies  upon  a  contractor’s 
discretion.  The  DOD  provides  oversight  by  using  risk-based  assessments.  Dodaro  (2013) 
pointed  out  that  non-DOD  industries  followed  a  standard  and  more  knowledge-based 
approach,  confirming  production  readiness.  As  a  result,  the  GAO  recommended  that  the 
DOD  deploy  a  more  disciplined  approach  to  manufacturing  development  and  production 
approval  pointing  to  the  success  in  non-DOD  industries. 

The  GAO  findings  provided  motivation  for  this  research,  which  examined  the 
DOD  acquisition  process  against  alternative,  non-DOD,  industrial  production  approval 
processes.  This  study  fills  a  significant  gap  in  the  research  with  respect  to  weapon 
systems’  poor  production  outcomes.  An  analysis  of  alternatives  (AO A)  comparing  the 
production  approval  processes  of  DOD  and  non-DOD  industries  identified  an  opportunity 

for  improvement  over  the  current-state  DOD  production  approval  process.  Key  findings 

xvii 


revealed  that  the  automotive  approach  followed  industry  best  practices  that  fully 
addressed  the  GAO  concerns  of  a  lack  of  manufacturing  knowledge  at  production  start. 
The  non-DOD  industries  used  a  more  prescriptive  set  of  quality  standards  that  enabled  a 
more  disciplined  development  and  demonstration  of  production  capability  prior  to 
production  commitment. 

The  AOA  relied  upon  the  identification  of  industry  best  practices.  A  set  of  high- 
level  stakeholder  needs  were  identified  and  related  to  a  set  of  best-practice  attributes  for 
production  approval  processes.  Characteristics  found  within  the  non-DOD  production 
approval  practices  were  not  observed  within  the  DOD  processes,  giving  an  indication 
why  non-DOD  industries  enjoyed  more  successful  production  outcomes: 

•  Third-party  compliance  to  quality  system  standards 

•  Prescriptive  advanced  quality  practices 

•  Common  quality  requirements  levied  to  the  entire  supply  network 

•  Knowledge-based  demonstration  of  production  capability 

•  Certification  warrant  demonstrating  production  readiness 

•  Quality  metrics  assessing  user  satisfaction 

Central  to  the  findings  in  support  of  the  study  AOA  was  an  assessment  of  two 
fundamental  types  of  Quality  Management  Systems  (QMS)  that  are  used  in  industry. 
There  were  remarkable  differences  in  production  success  depending  on  the  type  of  QMS 
used  by  an  organization.  While  all  industrial  sectors  incorporated  the  International 
Standards  Organization’s  (ISO)  ISO-9000  family  of  commercial  standards,  only  the 
automotive  sector  applied  a  more  prescriptive  quality  standard.  The  automotive  sector 
used  the  QS-9000  family  of  standards  published  by  the  Automotive  Industry  Action 
Group.  The  more  prescriptive  QMS  provided  structure  for  the  disciplined  development 
and  demonstration  leading  to  a  certification  of  production  capability.  The  QS-9000 
requirements  identified  by  an  automotive  original  equipment  manufacturer  (OEM)  were 
flowed  to  the  entire  automotive  supply  network.  In  contrast,  the  DOD  sector  was  found  to 


apply  the  ISO-9000  type  QMS  lacking  the  prescriptive  nature  of  the  automotive 
standards. 

There  were  three  alternative  industrial  sectors  examined  in  this  study.  All  three  of 
the  non-DOD  sectors  studied  used  a  certification  process  for  production  approval.  Only 
the  DOD  did  not  use  a  certification  process. 

Commercial  surveys  confirmed  the  benefits  of  the  prescriptive  quality  standards. 
The  benefits  reported  were  pivotal  to  this  study’s  preferred  solution  selection.  The 
adherence  by  suppliers  to  the  automotive  QMS  showed  improved  quality  and  better 
production  outcomes.  Data  obtained  from  the  Department  of  Transportation  supported 
the  benefits  in  product  reliability  growth  over  time  in  the  automotive  sector.  The 
assessment  of  alternate  production  approval  processes  provided  compelling  evidence  in 
favor  of  the  prescriptive  automotive  production  approval  process. 

An  example  that  could  be  followed  by  the  DOD  is  the  approach  taken  by  the 
automotive  industry  when  they  deployed  the  implementation  of  a  prescriptive  QMS 
process.  The  automotive  supply  base  used  the  services  of  the  Automotive  Industry  Action 
Group  (AIAG)  to  prepare  to  meet  new  quality  requirements.  The  AIAG  was  a  consortium 
formed  by  the  American  OEMs  to  communicate,  train  and  enable  the  implementation  of 
the  QS-9000  set  of  prescriptive  standards. 

The  automotive  QS-9000  standard  (later  renamed  ISO/TS- 16949)  is  consistent 
with  the  DOD’s  requirement  to  use  voluntary  consensus  standards  and  is  identified  in  the 
Federal  Acquisition  Regulations  (FAR  §46.202-4).  A  new  aerospace  standard  released  by 
the  Society  of  Automotive  Engineers,  AS-9145,  describes  an  advanced  product  quality 
planning  and  production  part  approval  process.  This  standard  captures  the  intent  of  the 
automotive  production  approval  process  available  for  DOD  application. 

It  is  the  recommendation  of  this  research  that  the  DOD  adopt  the  automotive 
production  approval  approach.  The  automotive  QMS  success  experience  can  be  applied 
to  the  DOD  to  reduce  acquisition  costs  and  improve  fleet  readiness.  A  future  ideal-state 
operational  view  is  shown  in  Figure  2.  This  figure  highlights  the  integration  of  best 
practices  found  in  non-DOD  organizations.  The  non-DOD  industries  have  developed  a 


more  disciplined  manufacturing  model  with  product  and  process  verification  required 
prior  to  the  start  of  production. 
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INTRODUCTION 


A.  OVERVIEW 

National  security  requires  an  ability  to  develop  military  capability  in  a  complex 
political  and  economic  global  environment.  Given  the  nature  of  an  ever-changing  military 
threat  coupled  with  challenging  economic  pressures,  there  is  a  clear  need  to  ensure  that 
best-value  outcomes  occur  within  the  federal  weapon  systems  acquisition  process. 
However,  this  has  not  always  been  the  case.  Certain  manufacturing  readiness  issues  in 
particular  have  resulted  in  poor  production  outcomes.  An  analysis  of  alternatives  (AO A) 
to  investigate  Department  of  Defense  (DOD)  and  non-DOD  production  approval 
processes  may  identify  an  opportunity  to  improve  production  outcomes  in  defense 
acquisition.  The  Government  Accountability  Office  (GAO)  reporting  has  provided 
insight  towards  understanding  specific  deficiencies  found  in  weapons  system  acquisition. 

1.  Poor  Production  Outcomes 

Annual  GAO  reports  on  select  weapon  programs  show  a  persistent  and  troubling 
observation  revealing  poor  production  outcomes  in  DOD  acquisition  (Dodaro  2013). 
Dodaro  identified  three  reasons  behind  the  poor  production  outcomes:  1)  knowledge  gaps 
in  technology,  2)  design-instability  and  3)  knowledge  gaps  in  manufacturing.  Of  these 
three  causes,  only  knowledge  gaps  in  manufacturing  lacked  DOD  attention.  Additional 
details  reported  by  Dodaro  concerning  manufacturing  causality  of  poor  production 
outcomes  included:  1)  use  of  non-standard  processes,  2)  failure  to  identify  critical 
manufacturing  processes  and  3)  failure  to  apply  statistical  process  control  prior  to 
production  start  (2013,  170). 

According  to  Dodaro,  the  DOD  acquisition  process  lacked  a  standard  and 
systematic  knowledge-based  transition-to-production  (TTP)  in  support  of  a  production 
decision  occurring  at  Milestone  C  (MS  C).  Key  findings  in  the  2013  GAO  report  revealed 
that  the  acquisition  of  weapon  systems  often  fail  to  deliver  reliable  and  mature 
technologies  into  the  Production  and  Deployment  (PD)  phase  from  the  Engineering  and 
Manufacturing  Development  (EMD)  phase.  The  consequence  of  this  acquisition  approach 
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has  been  poor  production  outcomes  after  MS  C.  Such  outcomes  adversely  affect  cost, 
schedule,  performance  and  related  military  readiness  in  the  field  of  operations.  A  look  at 
the  manufacturing  causality  invites  further  review  (Figure  1). 


Outcomes 


DOD  Weapon  System  Acquisition  uses 
Statutory  and  Procedural  Checkpoints  to 
Confirm  Production  Readiness  that  relies 
on  Technological  Maturity,  Design 
Stability  and  Manufacturing  Risk 

Figure  1.  GAO’s  Three  Factors  Resulting  in  Poor  Production  Outcomes. 

Adapted  from  Dodaro  (2013). 
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2.  GAO  Questions  the  DOD  Production  Approval  Process 

The  GAO  found  prime  contractors  in  DOD  acquisition  lacked  a  standard 
approach  to  achieve  a  stable  and  mature  production  process  at  the  MS  C  decision  point. 
In  2002,  the  GAO  had  reported  their  observation  that  there  was  a  lack  of  a  systematic  and 
standard  knowledge-based  production  readiness  approach  in  DOD  acquisition  (Schinasi 
2002).  This  lack  of  discipline  was  not  characteristic  of  other  industries  the  GAO 
reviewed.  These  findings  led  the  GAO  to  recommend  that  the  DOD  look  to  other 
industries  in  hopes  of  finding  improvement  opportunity. 

The  DOD  acknowledged  the  GAO’s  finding  that  there  were  no  standard  methods 
followed  related  to  manufacturing  development  and  production  capability  verification 
prior  to  MS  C.  This  finding  establishes  why  many  programs  fail  to  reach  performance 
goals  in  production.  One  characteristic  found  in  non-DOD  manufacturing  acquisition 
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processes  was  the  observation  of  a  more  disciplined  product  development  and  production 
demonstration  process  (Schwartz  2013,  27)  and  (USD[AT&L]  2015).  However,  there 
were  no  significant  policy  actions  taken  by  the  DOD  concerning  the  GAO’s  findings 
towards  a  more  knowledge-based  manufacturing  development  approach. 

3.  DOD  Acquisition  and  Production  Approval 

The  DOD’s  requirements  related  to  manufacturing  development  are  found  in  the 
Joint  Capabilities  Integration  and  Development  System  (JCIDS)  reference  manual.  The 
JCIDS  describes  requirements  on  how  a  program  development  is  conducted  and  outlines 
the  requirements  for  a  life  cycle  phased  acquisition  process.  With  respect  to 
manufacturing  development,  the  JCIDS  requires  a  risk-based  assessment  of  a  program’s 
manufacturing  maturity  but  does  not  require  an  actual  production  line  demonstration. 
According  to  the  JCIDS  instruction,  a  program  manager  (PM)  will  report  manufacturing 
risk  to  the  Milestone  Decision  Authority  (MDA)  through  the  capabilities  production 
document  (CPD)  at  MS  C.  However,  if  the  risk-based  assessment  does  not  rise  to  a 
significant  level  of  concern  a  PM  is  not  required  to  report  on  manufacturing  risk  (JCS 
2015b). 

In  contrast,  non-DOD  industry  practices  use  a  disciplined  knowledge-based 
production  approval  process  that  demonstrates  a  manufacturing  capability  based  on 
product  from  an  actual  production  line  to  report  product  and  process  capability  running  at 
production  rates.  As  such,  it  is  easy  to  see  a  stark  difference  contrasting  the  DOD’s  risk 
assessment  approach  that  does  not  rely  upon  an  actual  production  line  experience  to 
assess  production  capability  to  show  readiness  to  produce  (USD[AT&L]  2015,  26-29, 
84).  This  is  why  the  GAO  made  recommendations  to  consider  a  knowledge-based 
production  capability  demonstration  approach  to  determine  production  manufacturing 
readiness  at  the  end  of  the  EMD  phase  (Dodaro  2013). 

Defense  acquisition  using  the  JCIDS  calls  out  a  manufacturing  risk  assessment 
tool  to  evaluate  a  product’s  production  environment  to  measure  manufacturing  readiness 
level  (MRL).  The  establishment  and  maintenance  of  the  MRL  tool  is  directed  by  the 
Office  of  the  Secretary  of  Defense  (OSD)  within  the  OSD’s  Department  of  Research  & 
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Engineering’s  DOD  Manufacturing  Technology  (MANTECH)  (OSD  2012).  The  MRL 
procedure  guides  an  evaluation  that  interrogates  a  contractor’s  production  environment’s 
maturity.  Manufacturing  assessments  in  this  approach  fail  to  identify  actual  production 
line  performance  capability.  Therefore,  the  MRL  risk  assessment  provides  little  insight 
into  critical  manufacturing  data  to  support  production  decisions. 

Additional  guidance  within  the  defense  acquisition  system  (DAS)  related  to 
manufacturing  development  and  approval  processes  are  found  in  the  DOD  Instruction 
(DODI)  DODI-5000.02  (USD[AT&L]  2015).  The  DODI  and  JCIDS  guidance  falls  far 
short  of  the  discipline  of  non-DOD  organizations  where  there  is  a  requirement  to  validate 
an  actual  production  capability.  Of  concern  is  the  absence  of  manufacturing  process 
capability  data  in  MRL  risk  assessment  reporting.  A  manufacturing  readiness  assessment 
does  not  rely  upon  a  demonstration  of  production  capability.  Contractor  fabricated 
product  built  in  support  of  EMD  only  requires  a  contractor  assert  that  a  production 
environment  is  “production  relevant.”  This  MRL  assessment  of  relevant  does  not  mean 
that  that  the  early  production  or  fabrication  of  product  for  Low-Rate  Initial  Production 
(LRIP)  or  Pull  Rate  Production  (PRP)  is  fully  defined  and  validated. 

The  DOD  acquisition  guidance,  described  in  the  Defense  Acquisition  Guidebook 
(DAG)  on  systems  engineering  (SE),  stated  that  use  of  the  OSD  MRL  Guidebook  is  but 
“one  way”  to  assess  manufacturing  risk  (USD[AT&L]  2013,  chapter  4).  The  DAG  did 
not  go  on  to  discuss  any  other  ways  of  identifying  manufacturing  risk  or  point  to  any 
other  best  practices  to  inform  how  a  program  should  assess  production  readiness.  In  DOD 
acquisition,  the  decision  to  go  into  a  production  contract  allows  a  contractor  to  continue 
manufacturing  development  in  PD  with  inherent  manufacturing  risk.  Even  if  EMD 
fabricated  items  are  successfully  qualified  to  functional  requirements  little  is  known 
about  an  intended  production  line  definition  or  its  degree  of  manufacturing  robustness  in 
sustaining  as-built  product  requirements. 

The  JCIDS  DODI  5000.02  and  DAG  confirm  this  minimalist  approach  to 
production  definition  and  reporting  of  manufacturing  risk  (USD[AT&L]  2013).  The 
DOD  will  conduct  its  last  technical  review  in  EMD  as  a  final  assessment  of  production 
readiness  before  PD.  The  production  readiness  review  (PRR)  is  an  assessment  that  guides 
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this  final  inquiry  based  upon  a  checklist  to  measure  program  and  manufacturing  risk.  This 
PRR  is  part  of  a  program’s  reporting  to  the  MDA  at  MS  C  to  gain  approval  for  a 
production  go-ahead.  Early  production  will  start  with  manufacturing  development 
incomplete.  During  PD  is  when  a  contractor  finally  completes  development  of  the 
manufacturing  environment  for  production  where  the  government  acquisition  process 
attempts  to  confirm  production  capability.  The  post-production  assessment  is  a  technical 
review  called  a  Physical  Configuration  Audit  (PC A). 

The  PCA  is  a  standard  DOD  practice  that  occurs  after  LRIP  and  prior  to  Full-Rate 
Production  (FRP).  The  PCA  event  is  used  to  legally  define  the  saleable  production 
baseline  of  the  configured  item  for  on-going  production.  According  to  the  JCIDS,  the 
PCA  serves  as  a  graduation  event  to  enter  FRP  (USD[AT&F]  2015,  29). 

4.  The  Study  Production  Approval  Process  AOA 

The  AOA  conducted  in  this  study  relied  upon  the  identification  of  industry  best 
practices  related  to  manufacturing  development  and  process  verification.  A  set  of  high- 
level  stakeholder  needs  were  identified  and  related  to  a  set  of  best  practice  attributes  as 
used  in  disciplined  production  approval  processes.  Certain  production  approval  process 
characteristics  found  within  non-DOD  production  approval  practices  were  not  observed 
within  the  DOD  processes.  The  additional  production  approval  practices  found  in  non- 
DOD  industries  provided  an  indication  why  non-DOD  industries  enjoyed  more 
successful  production  outcomes.  Some  of  these  differences  in  non-DOD  organizations 
included: 

•  Third-party  compliance  to  quality  system  standards 

•  Prescriptive  advanced  quality  practices 

•  Common  quality  requirements  levied  to  the  entire  supply  network 

•  Knowledge-based  demonstration  of  production  capability 

•  Certification  warrant  demonstrating  production  readiness 

•  Quality  metrics  used  to  assess  user  satisfaction 
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B.  CONTEXT  OF  THE  PROBLEM 


The  problem  space  of  poor  production  outcomes  related  to  manufacturing 
causality  can  be  defined  through  the  audit  reports  of  the  GAO,  a  review  of  the  literature 
related  to  weapons  system  acquisition,  DOD  policy,  DOD  workforce  capability  and  DOD 
actions  that  try  to  improve  production  outcomes.  One  observation  within  the  DOD 
acquisition  policy  shows  favoritism  toward  product  performance  development  and 
verification  and  fails  to  provide  the  necessary  guidance  concerning  knowledge-based 
manufacturing  readiness.  This  is  particularly  significant  at  MS  C  where  manufacturing 
readiness  is  assessed  but  production  capability  is  not  assessed.  This  failure  to  require  a 
demonstrated  production  capability  prior  to  MS  C  coupled  with  a  contractor’s  lack  of 
discipline  to  develop  key  manufacturing  knowledge  during  EMD  transfers  manufacturing 
risk  into  LRIP/FRP. 

One  of  the  contributing  factors  related  to  poor  production  outcomes  is  DOD 
workforce  competency.  The  lack  of  a  broadly  skilled  workforce  hinders  the  ability  to 
develop  and  assess  production  maturity  and  capability.  This  occurs  in  DOD  contracting 
and  oversight  because  manufacturing  expertise  is  not  fully  staffed  or  understood  within 
DOD  SE  community.  As  such,  the  DOD  lacks  a  standard  and  systematic  method  in 
developing  manufacturing  maturation  and  has  failed  to  require  verified  process  stability 
and  control  (Sullivan  2008).  The  DOD’s  lopsided  focus  on  product  over  process 
performance  can  also  be  seen  by  a  review  of  the  many  product  performance  related 
technical  reviews  in  acquisition  (see  Figure  2)  (USD[AT&L]  2013).  Conversely,  non- 
DOD  organizations  are  staffed  to  develop  and  manage  their  manufacturing  development 
that  requires  a  certification  of  readiness  prior  to  entering  production. 
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AOTR  •  Assessment  of  Operational  Test 
Readiness 

ASR  -  Alternative  Systems  Review 
CDR  ■  Critical  Design  Review 
EMD  -  Engineering  and  Manufacturing 
Development 

FCA  -  Functional  ConfigurationAudit 

FD-  Full  Deployment 

FOC  •  Full  Operational  Capability 

FRP  -  Full-Rate  Production 

IOC  •  Initial  Operational  Capability 


ISR  -  In-Service  Review 
MDD  •  Materiel  Development  Decision 
OTRR-  Operational  Test  Readiness 
Review 

PCA  -  Physical  Configuration  Audit 
PDR  -  Preliminary  Design  Review 
PRR  -  Production  Readiness  Review 
S&T  -  Science  and  Technology 
SRR  •  System  Requirements  Review 
SFR  •  System  Functional  Review 
SVR  -  System  Verification  Review 
TRR  -  Test  Readiness  Review 


o 


Mandatory  technical  reviews 


/  \ 

■  /  Best  practice  technical  reviews  and 
audits 


v 


/  Testreviews(seeDAG Chapter!)) 


Figure  2.  Life  Cycle  Development.  Adapted  from  (USD[AT&L]  2013). 


The  weakness  related  to  a  lack  of  a  skilled  professional  manufacturing  oversight 
can  be  traced  to  a  memo  (Perry  1994)  that  eliminated  most  prescriptive  military  standards 
(see  “Appendix  A.  Perry  Memo”).  In  1994,  the  DOD  moved  to  a  Performance  Based 
Acquisition  (PBA)  practice  setting  in  motion  the  use  voluntary  consensus  standards. 
When  the  PBA  practice  commenced,  the  DOD  divested  itself  of  the  skills  to  assess 
integration  and  manufacturing  development.  Recently,  the  DOD  recognized  this  loss  of 
know-how  and  initiated  a  hiring  of  20,000  acquisition  professionals  by  the  end  of  fiscal 
year  2015  (Erwin  2010).  In  National  Defense  Industrial  Association  (NDIA)’s  Sandra 
Erwin’s  blog,  there  was  a  posting  concerning  the  2009  Weapon  Systems  Acquisition 
Reform  Act  (WSARA)  Progress  Report.  Here,  Erwin  chronicled  GAO  Director 
Sullivan’s  response  to  a  question  from  the  House  Oversight  and  Government  Reform 
chairman,  Rep.  John  Tierney,  D-Mass: 

The  law  alone  won’t  result  in  substantial  change  unless  there  are  leaders 
within  the  department  enforcing  it,  he  [Tierney]  said.  “It  will  take 
considerable  and  sustained  leadership  and  effort  to  change  the  incentives 
and  inertia  that  reinforce  this  status  quo.  And  I  think  the  Congress  has  a 
role  in  that  as  well.” 

Sullivan  then  posed  an  obvious  and  uncomfortable  question:  “One  has  to 
ask  why  extraordinary  actions  are  needed  to  force  practices  that  should 
occur  normally.”  Realistic  cost  estimates  and  assessments  of  the  maturity 
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of  technology  should  have  been  part  of  the  standard  modus  operandi  and 
not  required  major  legislation,  he  noted. 

Much  of  the  blame  for  the  shocking  cost  overruns  and  performance  of 
major  programs  has  been  laid  on  the  shortage  of  contracting  personnel  and 
technical  experts  at  the  Defense  Department.  WSARA  reinforces  the  issue 
and  calls  for  the  Pentagon  to  beef  up  its  in-house  skills.  Defense  Secretary 
Gates  last  year  announced  plans  to  hire  20,000  acquisition  professionals 
(ID  134). 

This  communication  captured  by  Erwin  appears  to  identify  one  concern  at  the  root 
of  the  manufacturing  knowledge  gap  reported  by  the  GAO:  manufacturing  expertise 
within  the  government  acquisition  process  lacks  experienced  personnel  resources.  If 
fortified  with  manufacturing  expertise,  then  there  would  likely  be  improved  oversight  and 
ability  to  assess  a  contractor’s  manufacturing  maturity  accurately.  In  the  short  term,  the 
DOD  had  developed  a  set  of  expert  questions  in  the  MRL  and  PRR  checklists  and 
communicated  them  to  the  combatant  commanders  (COCOMs)  and  PMs. 

The  release  of  the  DOD  MRL  guidebook  indicated  that  the  DOD  was  not  satisfied 
with  its  current  state  of  problems,  in  part,  due  to  manufacturing  issues  (OSD  2012).  The 
MRL  process  came  about  the  same  time  as  the  WSARA  hiring  initiative  in  hopes  that  the 
assessment  tool  would  improve  manufacturing  readiness  and  better  document 
manufacturing  risk.  Unfortunately,  the  GAO  findings  since  2010  still  discuss  the  problem 
of  poor  production  outcomes  from  manufacturing  workforce  deficiencies.  The 
manufacturing  knowledge-gap  problem  is  then  related  to  the  acquisition  process  and 
workforce  expertise. 

The  DOD  also  took  steps  to  augment  manufacturing  expertise  by  the  development 
of  centers  of  manufacturing  excellence.  One  center  setup  is  the  Department  of  Defense 
Research  and  Engineering  (DODR&E).  In  support  of  the  MRL,  the  DODR&E  discussed 
the  need  to  apply  the  MRL  process  in  a  development  program  (Dunn  2010).  This 
recommendation  was  implemented  by  the  DOD  and  is  now  specified  as  a  required 
practice  in  the  2015  JCIDS. 

Another,  but  long-standing,  center  of  excellence  developed  was  the  NDIA  as  a 
partnership  between  the  United  States  government  and  the  defense  industry.  In  a  2010 
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report  on  Air-Launched  Tactical  Weapons,  the  NDIA  called  for  a  certification  of  a 
product’s  production  readiness  (NDIA  Gulf  Coast  Chapter  2010,  19).  It  would  be  a  novel 
approach  for  the  DOD  to  use  a  knowledge-based  certification  for  manufacturing 
readiness  as  recommendation  by  the  NDIA. 

The  DOD  acquisition  guidance  initiative  to  monitor  how  a  production  line 
develops  and  matures  using  the  MRL  risk  assessment  process  showed  increased  attention 
given  to  the  manufacturing  competency  yet  these  actions  only  served  to  continue  the 
DOD’s  risk-based  approach  to  manufacturing  readiness.  The  DOD’s  lack  of  response  to 
the  NDIA  and  GAO  recommendations  related  to  manufacturing  improvement 
opportunity  ignored  the  value  placed  on  process  verification  used  by  non-DOD 
industries.  The  applications  of  best  practices  are  the  disciplined  focus  in  a  knowledge- 
based  manufacturing  development.  Failing  to  identify  and  improve  production  processes 
were  missed  opportunities  for  DOD  acquisition  to  reduce  problems  related  to  production 
risk.  Brock  and  Walker  pointed  out  that  product  knowledge  is  obtained  late  in  DOD 
acquisition  as  compared  to  organizations  that  follow  best  practices  in  non-DOD 
organizations  (see  Figure  3)  (Brock  2003;  Walker  2005). 


Production, 
design  &. 
technology 
maturity 


Design  & 
technology 
matu  rity 


T ec  hno  I  ogy 
matu  rity 


Figure  3.  Risk  Vice  Production  Knowledge.  Adapted  from  Walker  (2005). 
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1.  Problem  Space  Source  Data  Robustness 

Confidence  in  background  information  related  to  the  problem  space  comes  from 
examining  the  reliability  of  the  reports  used  to  describe  DOD  acquisition  deficiencies. 
One  question  to  answer  was  the  degree  of  integrity  of  the  findings  as  published  by  the 
GAO.  To  answer  this,  the  GAO  assured  integrity  based  on  long-standing  ethics  they 
apply  to  their  investigative  reporting  (see  “Appendix  B’’).  An  additional  strength  in  the 
quality  of  the  information  dealing  with  the  problem  space  definition  was  found  in  the 
GAO’s  reporting  and  the  DOD  responses  to  GAO  recommendations.  In  one  responsive 
letter,  the  DOD  took  some  exception  to  the  GAO’s  findings  but  indicated  that  the  DOD 
would  conduct  their  own  report  on  the  issues  discussed.  See  Appendix  G  -  DOD  Memo 
to  GAO.  At  times,  the  GAO  reports  discussed  how  the  DOD  implemented  or  planned  to 
implement  various  actions  to  satisfy  the  GAO  recommendations.  The  GAO  would 
provide  a  status  in  the  following  year’s  report  if  the  DOD  acted  upon  any 
recommendation.  Therefore,  these  periodic  reports  provided  a  continuum  of  objective 
observations  on  issues  identified  through  the  GAO  reporting. 

Consider  the  problem  space  represented  by  manufacturing  deficiencies  resulting 
from  a  lack  of  manufacturing  requirements  definition.  Since  manufacturing  development 
has  not  had  much  attention  in  DOD  acquisition,  there  is  an  omission  of  process  capability 
requirements  definition.  Non-DOD  manufacturing  development  follows  an  approach  that 
prevents  poor  production  outcomes  by  addressing  manufacturing  requirements  early. 

Objective  manufacturing  and  quality  requirements  in  non-DOD  organizations  are 
typically  defined  through  customer  or  regulatory  guidance  and  is  explicitly  expressed  in 
the  execution  of  an  organization’s  quality  management  system  (QMS).  Adhering  to  the 
practices  outlined  in  a  QMS  helps  define  a  contractor’s  business  practices  related  to 
manufacturing  and  quality  policy.  DOD  acquisition  practices  have  customarily  invoked 
the  International  Standards  Organizations  ISO-9000  or  similar  AS-9000  family  of  quality 
standards  that  do  not  require  customer  or  regulatory  requirements  to  be  defined.  The 
automotive  supply  base  followed  specific  customer  requirements  given  in  the  QS-9000 
QMS  published  by  the  Automotive  Industry  Action  Group  (AIAG).  The  AIAG  has  been 
a  key  enabler  supporting  the  deployment  of  the  automotive  QMS.  The  AIAG  served  the 
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automotive  supply  network  as  a  consortium  formed  by  the  American  original  equipment 
manufacturers  to  communicate,  train  and  enable  the  implementation  of  the  QS-9000  set 
of  prescriptive  standards. 

The  ISO  technical  specification  (TS)  ISO/TS- 16949  (successor  to  the  original  QS- 
9000  QMS)  is  related  to  the  automotive  industry.  It  is  interesting  that  the  CFR  references 
these  QMS;  however,  the  DOD  has  not  taken  an  active  role  to  invoke  these  more 
disciplined  QMS  (Appendix  H  -  CFR,  Title  48,  46.202.4)  and  (Walker  2006).  In  addition, 
the  National  Institute  of  Standards  reported  on  the  problem  space  with  its 
recommendation  to  use  the  automotive  QMS  (Breitenberg  1999).  While  these  references 
indicate  that  DOD  could  elect  to  include  production  requirements  characteristic  of  the 
automotive  prescriptive  standards  it  is  not  followed.  The  DOD  tends  to  shy  away  from 
how-to  requirements  based  upon  acquisition  practices  since  the  Perry  memo  giving 
attention  to  product  performance  rather  than  manufacturing  development  and  process 
capability  verification. 

Even  with  several  initiatives  in  workforce  development,  the  DOD  does  not  seem 
to  address  the  root  cause  of  poor  production  results  related  to  manufacturing.  The  DOD 
had  not  required  contractors  to  follow  a  disciplined  development  of  manufacturing 
requirements  likely  due  to  the  absence  of  an  accepted  prescriptive  standard.  Assessments 
of  risk  to  determine  the  state  of  readiness  to  produce  did  not  lend  itself  to  process 
improvement  for  manufacturing  capability  in  defense  acquisition.  Even  contractual 
incentive  type  contracts  have  been  problematic  to  improve  the  state  of  contracting  or  in 
validation  of  manufacturing  development  and  capability  knowledge.  According  to  one 
review  of  incentive  based  contracts  there  has  been  an  $8-billion-dollar  giveaway  not 
achieving  results  (Calvaresi-Barr  2005). 

1.  Supporting  Data  Concerning  Poor  Production  Outcomes 

Given  that  there  is  a  lack  of  manufacturing  knowledge  at  production  decision  in 
the  DOD  acquisition  process,  what  impact  is  there  on  fleet  readiness?  This  question  can 
be  answered  by  the  examination  of  several  reports  and  problem  related  datasets 
concerning  field  performance  that  gives  attention  to  the  size  of  the  problem. 


11 


First,  the  problem  space  of  poor  production  related  to  manufacturing  and  quality 
are  chronicled  in  a  GAO  report  to  Congress  that  reviewed  11  major  weapon  systems 
(Figure  4)  (Sullivan  2008).  The  Senate  also  looked  for  answers  by  asking  the  GAO  to 
consider  a  review  of  best  practices  in  2002  and  to  look  for  alternatives  to  improve 
(Schinasi  2002).  The  highlights  section  of  the  GAO  report  by  Sullivan  (2008)  spoke  to 
these  type  results  with  industries  outside  of  the  DOD  stating:  “In  contrast,  leading 
commercial  companies  GAO  contacted  use  more  disciplined  systems  engineering, 
manufacturing,  and  supplier  quality  practices.”  In  summary,  the  most  meaningful  way  to 
look  at  the  DOD  product  approval  process  problem  space  is  by  examining  the  GAO 
findings  within  their  basis  for  improvement  recommendations.  Interestingly,  the  GAO 
recommendations  did  not  provide  a  clear  path  for  improvement. 


System 

Source  of  quality  problem 

Impact  of  quality  problem 

Systems 

engineering 

Manufacturing 

Supplier  quality 

Cost 

(dollars  in  millions) 

Schedule 

Advanced  SEAL  Delivery 
System3 

y 

y 

$87 

Program  halted 

Advanced  Threat  Infrared 
Countermeasure/ 

Common  Missile  Warning 
System 

y 

y 

$117 

5-year  delay 

Expeditionary  Fighting 
Vehicle 

■S 

$750 

4-year  extension 
to  system 
development 

F-22A 

BS 

y 

y 

y 

$400 

No  schedule 
impact  to  program 

Global  Hawk3 

y 

y 

$239 

4-month 

production  slip  for 
sensor  suite 

Joint  Air-to-Surface 

Standoff  Missile 

y 

y 

$39 

Program  deferred 

LPD  17  Amphibious 
Transport  Dock3 

■S 

y 

y 

$846 

3-year  delay 

MH-SOs  Fleet  Combat 
Support  Helicopter 

y 

No  cost  impact 
to  program 

6- month 
production  slip 

Patriot  Advanced 
Capability-3 

y 

y 

y 

$26 

6-month  delay 

V-22  Joint  Services 
Advanced  Vertical 

Lift  Aircraft 

y 

$165 

Flight  operations 
halted  for 

1 7-months 

Wideband  Global 

SATCOM 

V 

y 

y 

$io 

1  8-month  delay  for 
initial  operating 
capability 

Figure  4.  Weapon  System  Quality  Problems.  Source:  Sullivan  (2008). 


Second,  the  Department  of  the  Navy  (DON)  Red  Stripe  data  that  tracks  failures 
that  ground  the  fleet  and  data  provided  by  the  Defense  Science  Board  (DSB)  concerning 
ARMY  programs  failing  to  meet  reliability  targets  measures  the  problem  scale  (DSB 
2008). 
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The  Red  Stripe  data  set  presents  a  proportional  contribution  of  failure  causes  from 
the  individual  failure  events  that  grounded  aircraft.  See  Figure  5.  The  data  segregates 
causal  factors  into  five  categories  on  a  percentage  basis:  manufacturing/quality  (33.6%), 
age/fatigue  (27.3%),  design  (12.5%),  maintenance  (22.4%)  and  those  not  yet  determined 
(4.2%). 


Causal  Factors  Assigned  to  RED  Stripe  Events 


4.2% 


■  Manufacturing  and 
Quality 

■  Age/Fatigue 

■  Design 

■  Maintenance 

mTBD 


■  Total 


Figure  5.  Causal  Factors  Assigned  to  Red  Stripe  Events. 


ARMY  development  program  results  provided  by  the  DSB  Taskforce  on 
Development  Test  and  Evaluation  (DT&E)  “Final  Report  of  DSB  Taskforce  on  DT&E” 
observed  that  almost  two  thirds  of  the  programs  monitored  fell  below  planned  reliability 
performance.  Reliability  is  expressed  as  an  average  such  as,  mean-time-between_ 
(MTB_).  Here,  MTB_  is  a  measure  such  as  the  mean  time  between  -  operational  failures 
expressed  in  usage  units.  See  Figure  6.  These  are  alarming  statistics. 
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Demonstrated  Reliability  vs  Requirements  for  All 
Operational  Tests 


Figure  6.  Demonstrated  Reliability  versus  Requirements.  Source:  DSB  (2008). 

2.  Risk  and  Knowledge-Based  Decision  Models 

Government  Accountability  Office  reporting  has  characterized  poor  production 
outcomes  due  to  an  acquisition  approach  that  focuses  on  risk  over  knowledge.  This 
bounds  the  GAO  argument  as  they  make  assertions  about  the  DOD’s  need  for 
improvement.  Since  the  GAO  looked  at  better  performing  production  outcomes  in  non- 
DOD  industries  it  became  important  to  focus  on  manufacturing  and  production  approval 
processes  in  those  industries.  The  primary  conclusion  of  the  GAO  involved  the  need  for  a 
more  knowledge-based  decision  process. 


a.  Contrasting  Risk  and  Knowledge-Based  Decisions 

When  the  GAO  discusses  a  knowledge-based  approach  to  characterize  the  success 
of  non-DOD  producers,  it  is  useful  to  review  what  they  meant  by  knowledge-based 
decisions  (Figure  7).  Between  the  years  1996  and  2013  the  GAO  developed  its 
assessment  framework  consistent  with  its  former  findings.  Later  reporting  by  the  GAO 
added  specific  criteria  to  assess  DOD  programs  against  certain  non-DOD  best  practices. 
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Knowledge  Point  Observations 

1 .  Many  of  the  programs  that  began  in  the  last  5  years  had  mature  technologies  and 
held  a  preliminary  design  review  prior  to  the  start  of  development  (knowledge  point 
1),  providing  a  better  foundation  to  avoid  future  cost  and  schedule  problems. 

2.  Less  than  one-third  of  the  programs  that  provided  data  on  design  drawings  released 
actually  reported  having  a  stable  design  at  their  critical  design  review  (knowledge 
point  2),  and  the  use  of  other  knowledge-based  practices  to  ensure  design  stability 
at  this  critical  juncture  varied. 

3.  Many  of  the  programs  we  assessed  have  taken  or  plan  to  take  steps  to  capture 
critical  manufacturing  knowledge  prior  to  the  start  of  production  (knowledge  point  3), 
although  the  methods  used  varied. 


Figure  7.  Knowledge  Point  Observations.  Source:  Dodaro  (2013). 


The  GAO  reported  that  non-DOD  programs  followed  a  disciplined  production 
development  and  process  demonstration  practice  that  achieved  better  product  reliability. 
The  GAO  indicated  that  non-DOD  best  practices  required  a  demonstration  of  production 
capability  using  statistical  control  prior  to  production  commitment.  The  GAO  defined 
three  knowledge  points  in  their  weapons  program  assessment  framework.  Knowledge 
point  three  was  concerned  with  manufacturing  (Dodaro  2014)  (Figure  8). 


Knowledge  Point  3:  Production  meets  cost,  schedule,  and  quality  targets.  Decision 
to  produce  first  units  for  customer 

Demonstrate  manufacturing  processes 

Build  and  test  production-representative  prototypes  to  demonstrate  product  in  intended 
environment 

Test  production-representative  prototypes  to  achieve  reliability  goal 
Collect  statistical  process  control  data 

Demonstrate  that  critical  processes  are  capable  and  in  statistical  control 

Conduct  independent  cost  estimate 

Conduct  independent  program  assessment 

Conduct  major  milestone  decision  review  to  begin  production 


Figure  8.  Knowledge  Point  3  Criteria.  Source:  Dodaro  (2014). 

The  Acquisition  Community  Connection  website  managed  by  the  Defense 
Acquisition  University  (DAU)  provided  a  definition  of  a  knowledge-based  acquisition 
approach  (DAU  2015a): 

Knowledge-Based  acquisition  is  a  management  approach  which  requires 
adequate  knowledge  at  critical  junctures  (i.e.,  knowledge  points) 
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throughout  the  acquisition  process  to  make  informed  decisions.  DOD 
Directive  5000.1  calls  for  sufficient  knowledge  to  reduce  the  risk 
associated  with  program  initiation,  system  demonstration,  and  full-rate 
production.  DOD  Instruction  5000.2  provides  a  partial  listing  of  the  types 
of  knowledge,  based  on  demonstrated  accomplishments  that  enable 
accurate  assessments  of  technology  and  design  maturity  and  production 
readiness  (ID  24660). 

and 

El.  14.  Knowledge-Based  acquisition.  PMs  [Program  Managers]  shall 
provide  knowledge  about  key  aspects  of  a  system  at  key  points  in  the 
acquisition  process.  PMs  shall  reduce  technology  risk,  demonstrate 
technologies  in  a  relevant  environment,  and  identify  technology 
alternatives,  prior  to  program  initiation.  They  shall  reduce  integration  risk 
and  demonstrate  product  design  prior  to  the  design  readiness  review.  They 
shall  reduce  manufacturing  risk  and  demonstrate  producibility  prior  to 
full-rate  production  (ID  24660). 

A  knowledge-based  approach  related  to  manufacturing  readiness  assessment  in 
the  DOD  is  deferred  to  full-rate  production.  There  is  a  risk  versus  knowledge  struggle 
that  favors  risk  by  allowing  a  broad  flexibility  for  PMs  to  satisfy  programmatic 
manufacturing  risk  reporting  at  MS  C  in  DOD  acquisition.  The  current  direction  towards 
a  DOD  production  decision  uses  knowledge  points  to  assess  readiness  but  still  largely 
operates  as  if  it  is  a  risk-based  decision  process  contrary  to  the  updated  policy  and  DAG 
guidance.  With  a  lack  of  manufacturing  process  definition  and  a  lack  of  data  from 
process  controls  on  early  fabrication  of  product  in  EMD  it  is  unlikely  that  decision 
makers  will  have  assurances  for  successful  achievement  of  production  rates  and  a  desired 
production  capability  in  PD.  This  contributes  to  the  weakness  observed  by  the  GAO  in 
the  DOD  production  approval  process.  The  May  2015  MRA  Guidebook  (version  2.3) 
states  that 

the  final  stage  of  EMD  is  producing  products  that  look  and  operate  like 
they  are  production  units  from  LRIP.  These  units  need  to  be  built  on  a 
pilot  production  line  to  adequately  demonstrate  the  ability  to  migrate  from 
EMD  to  LRIP.  Without  this  realism  it  would  be  very  difficult  to  obtain 
confidence  that  the  production  process  will  be  able  to  meet  cost,  schedule, 
and  performance  (e.g.,  quality)  requirements  for  production  (15). 
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b.  GAO  Assesses  Contractor  Improvement  Effort 

The  GAO  (2013)  reported  that  contractors  did  improve  their  manufacturing 
readiness  by  demonstrating  manufacturing  product  knowledge.  However,  a  scorecard 
review,  as  published  by  the  GAO,  revealed  there  was  a  lack  of  process  control  and 
demonstration.  The  GAO  use  of  “scorecards”  failed  to  show  that  contractors  complied 
with  the  GAO’s  former  recommendations  for  a  statistical  demonstration  of  production 
processes  and  controls  across  the  programs  the  GAO  reviewed.  However,  the  GAO 
scorecards  did  not  fully  support  Dodaro’s  assessment  (2013)  with  the  findings  that  things 
were  improving  over  prior  years  with  respect  to  production  maturity  knowledge:  “Many 
of  the  programs  are  capturing  critical  manufacturing  knowledge  prior  to  production,  but 
their  methods  vary.” 

Fact  checking  the  scorecards  did  not  show  evidence  of  the  GAO’s  assertion  that 
programs  improved  over  the  prior  years.  What  was  observed  is  that  the  scorecards  refute 
these  claims — an  example  from  AIM  9X  Block  II’s  assessment  (2013).  The  repudiation 
of  “scorecard”  success  found  that  the  data  collected  indicated  that  a  contractor  would  be 
scored  favorably  by  the  GAO  when  audited  but  the  contractor  only  planned  to  control  the 
process  at  some  future  time  (Figure  9). 


Attainment  of  Product  Knowledge 


As  of  January  2013 


Resources  and  requirements  match 


•  Demonstrate  all  critical  technologies  in  a  relevant 
environment 

•  Demonstrate  all  critical  technologies  in  a  realistic 
environment 


•  Complete  preliminary  design  review 


Product  design  is  stable 


•  Release  at  least  90  percent  of  design  drawings 

•  Test  a  system-level  integrated  prototype 


Manufacturing  processes  are  mature 


•  Demonstrate  critical  processes  are  in  control 

•  Demonstrate  critical  processes  on  a  pilot  production  line 


o 


•  Test  a  production-representative  prototype 


Knowledge  attained 
o  Knowledge  not  attained 


■  ■■■  Information  not  available 

Not  applicable 


Figure  9.  Product  Knowledge  Score  Card.  Source:  Dodaro  (2013). 


17 


3. 


Problem  Statement 


National  defense  is  affected  by  any  reduction  in  fleet  readiness  caused  by  poor 
production  outcomes.  The  lack  of  a  production  capability  demonstration  at  production 
start  is  one  of  those  detractors.  The  DOD’s  inability  to  address  manufacturing  capability 
issues  reported  by  the  GAO  stems  from  policy  inadequacies  and  a  lack  of  a  standard 
means  to  assure  there  is  a  readiness  to  produce. 

4.  Research  Questions 

The  following  research  questions  helped  develop  the  study  problem  statement.  A 
listing  of  specific  research  questions  to  be  answered  by  this  study  follows: 

•  How  is  production  readiness  defined  in  DOD  acquisition? 

•  How  does  the  DOD  manage  manufacturing  maturity  and  risk  prior  to  MS 
C? 

•  Why  is  there  a  lack  of  production  readiness  evidence  for  a  decision  at  MS 
C? 

•  Are  non-DOD  production  approval  processes  more  successful  than 
comparable  DOD  processes  and  in  what  ways? 

•  How  can  an  alternative  production  approval  process  improve  success  in 
the  DOD  acquisition  environment? 

C.  PROJECT  OBJECTIVES 

The  deficiencies  in  DOD  production  outcomes,  identified  by  the  GAO’s  Dodaro, 
provided  a  motivation  for  this  research  that  intends  to  identify  a  more  successful  TTP  not 
found  in  the  current  state  of  the  DOD  production  approval  process  (2013).  Coming  at  the 
problem  space  from  many  directions  helped  identify  how  to  translate  that  discussion  into 
a  set  of  statements  representing  the  study  purpose: 

•  Identify  specific  deficiencies  and  causal  factors  for  poor  production 
outcomes  in  DOD  acquisition  practices 
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•  Provide  a  SE  approach  to  study  alternative  industry  production  approval 
processes  as  compared  to  DOD  production  approval  process 

•  Develop  assessment  criteria  for  an  analysis  of  alternatives  for  a  production 
approval  process 

•  Conduct  analysis  with  discriminating  measures  of  performance  attributes 

•  Report  to  stakeholders  the  findings  and  recommendations  from  this  study 

•  Identify  potential  future  research  to  exploit  the  research  conclusions 

D.  SCOPE 

While  the  GAO  pointed  to  three  high-level  factors  contributing  to  poor  production 
outcomes  in  DOD  acquisition  only  manufacturing  causal  factors  became  the  focus  of  this 
study.  This  study  found  that  there  had  been  little  research  addressing  the  acquisition 
consequences  due  to  the  lack  of  a  standard  approach  to  manufacturing  development  and 
demonstration.  An  examination  of  DOD  manufacturing  development,  readiness  and  the 
production  approval  process  created  the  domain  of  the  manufacturing  knowledge  gap 
pointed  out  by  the  GAO  (Table  1). 


Table  1.  Selecting  the  Study  Domain:  Manufacturing  Knowledge  Gaps 


Causal 
Factors  Poor 
Production 
Outcomes 

Research 

Novelty 

Trends  in  Acquisition 
Reform 

Study 

Significance 

Study 

Scope 

Technological 

Knowledge 

gaps 

Not  Novel,  many 
research 

publications,  one 
technical  journal 
alone  citing  32 
other  related 
research  items. 

DODI  5000.02  included 
significant  reforms  to 
address  this  gap.  GAO  has 
observed  more  successes  in 
recent  reviews.  Other  policy 
guidance  also  reviewed 
showing  reform  and 
improvement. 

Related,  but  not 
significant.  There 
is  a  high  degree  of 
independence  with 
respect  to 
technological 
maturity. 

Excluded 
(gray  row) 
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Causal 
Factors  Poor 
Production 
Outcomes 

Research 

Novelty 

Trends  in  Acquisition 
Reform 

Study 

Significance 

Study 

Scope 

Design- 

instability 

Not  Novel,  many 
research 
publications. 

DODI  5000.02  included 
significant  reforms  to 
address  this  gap.  GAO  has 
reported  more  successes  in 
recent  reviews.  One 
example  is  conducting  a 
Preliminary  Design  Review 
in  Technology  Maturation 
and  Risk  Reduction. 

Related,  but  not 
significant.  There 
was  a  high  degree 
of  independence 
with  respect  to 
technological 
maturity. 

Excluded 
(gray  row) 

Manufacturing 

Knowledge 

gaps 

Novel  study  field 
with  little  research 
outside  of  the  GAO 
addressing  the 
production 
approval  prior  to 

MS  C. 

GAO  reported  that  there 
was  a  lack  of  production 
knowledge  in  DOD 
acquisition  as  compared  to 
other  more  successful 
commercial  development 
programs. 

DOD  policy 
documents  and 
guidelines  in  2015 
do  not  require  a 
manufacturing 
demonstration  of 
capability  prior  to 
MS  C. 

Included 

E.  REPORT  ORGANIZATION 

This  report  is  organized  in  five  chapters  and  an  appendix.  The  remaining  chapters 
are  described  as  follows: 

Chapter  II  includes  the  background  section,  which  addresses  the  study  domain 
with  respect  to  SE  as  a  tool  to  analyze  alternatives  as  applied  to  production  approval 
processes  contrasting  DOD  and  non-DOD  industries.  The  literature  review  section 
develops  important  elements  of  the  study  context  related  to  production  approval 
processes  from  the  point  of  view  of  poor  production  outcome  causality  focusing  the  study 
on  the  issue  of  a  lack  of  manufacturing  knowledge. 

The  study’s  analytic  approach  is  discussed  in  Chapter  III.  Lower- level  details  for 
measurable  performance  used  in  the  AOA  are  developed.  An  operational  view  of  the 
problem  definition  and  role  of  stakeholders  provides  a  construct  that  models  the  current 
DOD  and  “Ideal  State”  based  on  industrial  sector  best  practices.  The  development  of  the 
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six-step  Systems  Engineering  Development  Process  (SEDP)  is  described  and  is  applied  to 
this  study. 

Chapter  IV  discusses  the  study’s  analytic  approach  using  the  measurable 
performance  evaluation  criteria  developed.  Commercial  surveys  investigated  confirmed 
the  benefits  of  the  prescriptive  quality  standards.  The  study  identified  a  preferred  solution 
selection. 

Chapter  V,  the  last  chapter  of  the  thesis,  discusses  the  findings,  summarizes  the 
results,  and  provides  recommendations.  In  addition,  there  is  a  brief  discussion  related  to 
an  opportunity  for  future  study.  Key  enablers  in  support  of  a  DOD  process  improvement 
are  highlighted. 

The  appendices  support  certain  complex  discussions  found  in  the  thesis.  The 
items  expand  an  understanding  for  practices  that  are  typically  unfamiliar  to  a  DOD 
acquisition  audience.  A  whitepaper  is  provided  that  treats  the  subject  of  a  DOD  technical 
warrant  for  use  in  any  implementation  of  a  standardized  advanced  product  quality 
planning  (APQP)  /  production  part  approval  process  (PPAP)  for  DOD  acquisition. 
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II.  BACKGROUND  AND  LITERATURE  REVIEW 


A.  BACKGROUND 

The  background  section  is  a  discussion  of  general  SE  principals  and  then  a  careful 
development  of  the  issues  in  the  problem  of  poor  production  outcomes  in  manufacturing 
causality  in  DOD  acquisition  practices.  This  requires  a  development  of  the  current- state 
of  the  DOD  production  approval  practices  and  its  shortcomings.  Insight  gained  by 
studying  the  DOD  process  of  manufacturing  development  provides  a  basis  for  evaluation 
for  process  improvement. 

B.  SYSTEM  ENGINEERING  METHODS 

Systems  engineering  methods  can  provide  a  problem  solving  framework  to 
examine  issues  surrounding  the  manufacturing  knowledge  gap  that  has  resulted  in  poor 
production  outcomes  in  DOD  acquisition.  In  this  study,  an  AOA  assessment  relied  upon 
the  selection  of  candidate  alternatives  and  the  development  of  assessment  criteria  that 
was  used  to  study  process  improvement  potential.  A  definition  of  systems  engineering  is 
a  starting  point  that  helped  define  the  problem  solving  process  of  the  study  AOA.  The 
development  and  application  of  the  AOA  model  lead  to  the  identification  of  a  coherent 
SE  approach  aiding  in  the  selection  of  a  preferred  solution. 

1.  General  Systems  Engineering  Definition 

The  developing  practice  of  SE  and  its  body  of  knowledge  includes  the  work  of  the 
International  Council  on  Systems  Engineering  (INCOSE).  Two  popular  definitions  of  SE 
help  to  provide  a  context  for  the  use  of  SE  for  problem  solving  related  to  a  process  as 
significant  as  production  approval  in  weapons  procurement.  According  to  INCOSE 
(2007), 

systems  engineering  is  an  interdisciplinary  approach  and  means  to  enable 
the  realization  of  successful  systems.  It  focuses  on  defining  customer 
needs  and  required  functionality  early  in  the  development  cycle, 
documenting  requirements,  and  then  proceeding  with  design  synthesis  and 
system  validation  while  considering  the  complete  problem.  Systems 
engineering  considers  both  the  business  and  the  technical  needs  of  all 
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customers  with  the  goal  of  providing  a  quality  product  that  meets  the  user 
needs  (introduction).  According  to  the  Defense  Acquisition  Guidebook 
(DAG), 

systems  engineering  is  an  interdisciplinary  approach  encompassing  the 
entire  technical  effort  to  evolve  and  verify  an  integrated  and  total  life  cycle 
balanced  set  of  system,  people,  and  process  solutions  that  satisfy  customer 
needs.  Systems  engineering  is  the  integrating  mechanism  across  the 
technical  efforts  related  to  the  development,  manufacturing,  verification, 
deployment,  operations,  support,  disposal  of,  and  user  training  for  systems 
and  their  life  cycle  processes.  Systems  engineering  develops  technical 
information  to  support  the  program  management  decision-making  process. 

(OSD  2009,  Chapter  4.0.2) 

From  these  definitions  there  is  a  need  to  apply  the  systems  engineering  discipline 
to  manufacturing.  A  manufacturing  process  definition,  like  a  product  definition,  can 
approach  manufacturing  development  as  a  series  of  problem  solving  activities  that 
identifies  process  functionality  in  relation  to  the  satisfaction  of  product  requirements. 
System  Engineering  works  to  evolve  a  preferred  production  solution  that  will  create  a 
product  that  reliably  meets  user  needs.  DOD  weapons  system  acquisition  does  include 
manufacturing  development  and  production  approval  process.  Manufacturing  engineering 
is  therefore  a  concurrent  partner  with  engineering  development  sharing  a  goal  of 
delivering  an  affordable  and  producible  engineered  design. 

2.  System  Engineering  Development  Process  (SEDP) 

The  SEDP  is  a  SE  problem-solving  approach  focusing  on  system  needs  and 
continues  toward  an  implementation  of  a  supporting  solution  (Gibson,  Scherer  and 
Gibson  2007,  29-34).  The  SEDP  approach  defines  an  ideal  (or  normative)  state  related  to 
a  desired  outcome.  In  production  approval  processes  the  normative  state  would  be  “good” 
production  outcomes.  Similarly,  the  SEDP  approach  assesses  the  as-is  descriptive  state  of 
the  DOD  production  approval  process  with  the  related  undesired  poor  production  outcome. 
An  excellent  graphical  illustration,  not  intended  to  show  details,  captures  visually  the  nature 
of  the  SEDP  approach  that  considers  product  or  process  alternatives  while  moving  the 
development  along  interpretively  to  a  solution  (Figure  10)  (Sullivan,  Broullette  and  Joles 
1998). 
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Figure  10.  System  Engineering  Design  Process  (SEDP). 

Source:  Sullivan,  Broullette  and  Joles  (1998). 


In  addition,  the  early  engineering  effort  characterizes  requirements  and  provides  a 
problem  definition  in  relation  to  the  goals  associated  with  that  of  an  ideal  solution.  The 
process  of  establishing  meaningful  criteria  used  to  assess  the  benefits  of  one  alternative 
over  another  can  involve  a  criteria  ranking  scheme  allowing  the  analysis  to  discriminate 
between  candidates.  The  resulting  assessment  uses  numerical  scoring  comparing 
alternatives  to  support  a  recommendation  towards  a  preferential  option  that  best  aligns 
with  stakeholder  needs  and  overall  program  goals. 

The  SEDP  can  be  further  described  as  a  sequential  method  of  iterating  around  a 
solution  space  over  variances  in  the  factors  of  analysis  moving  engineering  effort  towards 
a  preferred  solution.  See  Table  2.  Gibson  notes  that  this  approach  can  be  controversial  in 
its  rating  and  ranking  of  metrics  that  are  hard  to  characterize  for  comparison  purposes 
(Gibson,  Scherer  and  Gibson  2007,  33).  Inhibiting  issues  may  include  political 
implications  or  a  lack  of  sufficient  background  to  substantiate  the  impact  of  factors 
influencing  design  or  process. 
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Table  2.  Six  Steps  on  How  to  Do  a  Systems  Analysis 


Step 

1 

Determine  the  (values)  goals  of  the  system 

2 

Establish  criteria  for  rating  alternative  candidates 

3 

Identify  or  develop  candidate  alternative  solutions 

4 

Rank  alternative  candidates 

5 

Iterate  as  necessary 

6 

Action 

Adapted  from  Gibson,  Scherer  and.  Gibson  (2007). 


C.  DOD  ACQUISITION  POLICY  AND  PROCESSES 

The  lack  of  clear  policy  and  guidance  with  respect  to  manufacturing  development 
and  demonstration  in  DOD  weapons  acquisition  is  a  finding  frequently  identified  in 
GAO’s  annual  reports  to  Congress  (Dodaro  2013).  This  omission  of  clear  guidance  for 
manufacturing  demonstration  is  an  obstacle  to  realizing  the  benefits  of  best  practices 
found  in  non-DOD  industries. 

Prior  to  the  Perry  memo  military  standards  were  more  prescriptive  and  included 
manufacturing  requirements.  The  Office  of  Management  and  Budget’s  (OMB)  conducts 
annual  compliance  reporting  to  assure  military  standards  are  only  used  when  vital  (OMB 
1998).  The  dramatic  change  brought  on  by  the  Perry  memo  still  limits  the  use  of  how-to 
standards.  Consider  the  policy  statement  from  DOD  Instruction  No.  4120.24  (Breitenberg 
1999,  1): 

3.  POLICY.  It  is  DOD  policy  that: 

a.  The  Department  of  Defense  shall  maintain  a  single,  integrated  DSP 
[Document  Standardization  Program]  to  promote  standardization  of 
materiel,  information  technology,  facilities,  and  engineering  practices  in 
accordance  with  Reference  (c). 

b.  Non-government  standards  shall  be  used  in  preference  to  developing 
and  maintaining  Government  specifications  and  standards  as  required  by 
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section  12(d)  of  Public  Law  104-113  (Reference  (d)),  unless  they  fall 
under  one  of  the  exceptions  specified  in  section  12(d)  of  Reference  (d). 

In  the  Perry  approach,  there  is  the  reduced  use  of  military  standards  in  favor 
commercial  consensus  standards.  An  unintended  consequence  has  been  the  loss  of 
manufacturing  requirements  and  know-how  in  DOD  acquisition  (policy  and  workforce). 
Today,  there  are  no  suitable  consensus  standards  found  in  policy  guidance.  In  addition, 
there  is  a  legacy  resistance  to  change  current  policy  and  practices  given  the  established 
contractor-defined  performance-based  acquisition  approach.  The  omission  of 
manufacturing  requirements  in  development  programs  with  the  removal  of  prescriptive 
military  standards  gave  rise  to  the  use  of  non-standard  contractor  developmental 
practices.  The  various  contractor  approaches  to  manufacturing  development  drives 
uncertainty  into  DOD  oversight  and  inhibits  improvement  due  to  an  omission  of  standard 
practices  in  manufacturing  development  and  knowledge. 

The  DOD  has  been  slow  to  move  towards  defining  a  more  objective  treatment  of 
manufacturing  in  development.  For  example,  if  reference  is  given  to  the  DAU  definition 
of  “knowledge-based”  acquisition  then  there  should  be  a  discussion  and  policy  that 
supports  producibility  and  affordability  activities.  Producibility  is  a  design  activity 
primarily  concerned  with  making  an  item  more  affordable  based  on  a  design’s  ease  of 
manufacturing.  This  simple  statement  shows  how  interrelated  designing  and 
manufacturing  are  as  partners  in  development.  However,  the  DOD  guidance  documents 
do  not  standardize  or  require  a  prescriptive  process  of  manufacturing  development  and 
demonstration.  Manufacturing  development  lacks  requirements  specification  in  DOD 
acquisition  and  does  not  find  itself  as  an  equal  development  partner  with  product  design 
and  verification  requirements. 

Generally,  the  DOD  production  readiness  methods  provide  visibility  to  prime 
contractor  behaviors  with  limitations  on  oversight  into  the  supply  network.  There  exists  a 
contracting  challenge  in  DOD  acquisition  to  gain  insight  into  the  entire  supply  network. 
This  was  discussed  in  one  GAO  report  pointing  to  the  automotive  sector’s  QMS  approach 
that  is  flowed  to  all  tiers  in  the  supply  network  consistently.  It  was  not  clear  if  the  intent 
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of  this  reference  was  a  recommendation  to  apply  the  prescriptive  automotive  QMS  to  the 
entire  supply  network  as  a  requirement  (Schinasi  1996). 

Another  difference  illustrating  contrast  between  the  DOD  and  automotive  OEMs 
is  the  automotive  response  to  the  global  economic  environment  is  uninhibited  where  the 
DOD  prime  contractors  may  be  restrictive  in  global  sourcing  opportunities  given  obvious 
security  issues  in  defense  acquisition.  Such  examples  as  these  can  negatively  impact 
acquisition  outcomes,  but  they  would  be  exceptions  to  the  basic  approach  non-DOD 
manufacturers  follow  in  there  production  approval  processes. 

D.  DOD  PRODUCTION  DECISION  AND  ACQUISITION  POUICY 

The  DOD  acquisition  guidance  comes  from  four  main  policy  documents  and  one 
guidebook  used  principally  to  define  the  acquisition  approach  for  DOD  weapons 
acquisition.  Table  3  lists  each  of  these  key  documents  with  a  brief  description  of  the 
associated  sponsoring  organization,  document  number,  issuance  date,  and  a  comment 
derived  to  capture  the  purpose  of  the  document.  One  significant  finding  from  a  review  of 
these  policy  documents  is  that  they  describe  a  risk-based  assessment  and  not  a 
knowledge-based  demonstration  of  production  capability.  As  a  result,  programs  proceed 
into  production  with  significant  production  risk  entering  the  PD  phase.  Poor  production 
outcomes  are  the  undesired  consequence  given  the  findings  as  reported  by  the  GAO 
Dodaro  (2013).  The  understanding  conveyed  is  that  the  DOD  process  is  deferring 
development  of  critical  manufacturing  knowledge  by  waiting  until  after  early  production 
experience. 
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Table  3.  Primary  Governing  Policy  Documents  for  DOD  Acquisition 


Sponsor 

Document 

Number 

Date  of 

Issuance 

Document  Title 

Comment-Purpose 

Joint  Chiefs  of 
Staff 

CJCSI 

3170.011 

23-Jan-15 

Joint  Capabilities 
Integration  And 
Development 
System  (JCIDS) 

“The  JCIDS  process  provides 
organizations  with  the  guidance  and 
ability  to  validate  Capabilities 
documents:  ICD,  CDD,  CPD”  (2) 

Joint  Chiefs  of 
Staff 

JC1DS 

Manual 

12-Feb-15 

JCIDS  Manual 

“This  manual  provides  information 
regarding  activities  including  mandatory 
training  for  personnel  involved  in  the 
requirements  processes,  capability 
requirement  portfolio  management, 
identification  of  capability  requirements 
and  associated  capability  gaps, 
development  of  capability  requirement 
documents,  gatekeeping,  and  staffing 
procedures.”  (1) 

Department  of 
Defense 

DODD 

5000.01 

12-5-2003 
Cert  1  2  - 
Nov-2007 

DOD  Directive  - 

The  Defense 
Acquisition  System 

“The  Defense  Acquisition  system  is 
intended  to  acquire  quality  products 
satisfying  user  needs  that  achieve 
mission  capability  at  a  fair  and 
reasonable  price.”  (3) 

Department  of 
Defense 

DODI 

5000.02 

7-Jan-15 

DOD  Instruction  - 
Operation  of  the 
Defense 

Acquisition  System 

“The  overarching  management 
principles  and  mandatory  policies  that 
govern  the  Defense  Acquisition 

System...”  (1) 

Acquisition 
Technology  & 
Logistics 

DAG 

15-May- 

2013 

Defense 

Acquisition 

Guidebook 

“The  Defense  Acquisition  Guidebook  is 
intended  to  complement  policy 
documents  with  discretionary  best 
practice  that  should  be  tailored  to 
program  needs.”  (DAU  2015b, 
ID=654219) 

1.  Knowledge-Based  Acquisition  and  Technical  Authority 

The  DOD  acquisition  process  includes  program  authorities  that  are  the  actors  in  a 
program’s  management.  Recent  DOD  acquisition  reforms  expressed  in  the  DAS  describe 
the  intention  to  develop  a  weapons  system  using  a  stated  paradigm  shift  in  the  supported 
SE  approach  from  “risk-based”  management  to  a  “knowledge -based”  acquisition 
strategy.  Specifically,  there  is  a  discussion  of  a  basic  acquisition  approach  calling  for  a 
process  capability  assessment  aligned  to  the  new  knowledge-based  strategy.  While  noted 
in  the  DAG  the  knowledge-based  approach  is  not  operationalized  in  practice  in  DOD 
acquisition.  Consider  the  following  from  the  DAG: 
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Positive  acquisition  outcomes  require  the  use  of  a  knowledge-based 
approach  to  product  development  that  demonstrates  high-levels  of 
knowledge  before  significant  commitments  are  made.  In  essence, 
knowledge  supplants  risk  over  time  (chapter  4). 

An  example  brings  some  clarity  to  the  difference  between  knowledge-based  and 
risk-based  approaches  in  SE.  Manufacturing  risk  management  is  documented  through  a 
system  engineering  technical  review  (SETR)  that  is  accomplished  by  the  use  of  formal 
checklists  that  have  been  developed  as  guidance  in  support  an  acquisition  milestone 
review.  In  the  case  of  the  MRL  assessments,  they  are  not  used  as  entrance  or  exit 
requirements  at  acquisition  milestones.  Instead,  DOD  guidance  describes  manufacturing 
risk  assed  by  the  MRL  simply  as  a  status  report. 

In  addition  to  the  MRL  assessments  the  NAVY  will  often  use  the  OSD  SETR 
checklist  tool  to  report  overall  program  risk.  The  ARMY  supports  a  gated  SE  review 
process  with  a  checklist  called  the  Product  Assurance  Risk  Level  (PARL)  assessment. 
The  ARMY  Aviation  &  Missile  Research,  Development  &  Engineering  Center  maintains 
the  PARL  and  is  not  a  publicly  accessible  web  product.  In  each  case,  manufacturing 
reviews  consider  the  risk  to  manufacture  and  do  not  require  any  validation  or 
demonstration  of  process  capability  knowledge  prior  to  production  decisions.  These  risk 
assessments  do  not  follow  the  DAG  paradigm  shift  to  knowledge -based  acquisition. 

The  DAG  downplays  the  advocacy  of  the  MRL  checklists  within  the  policy 
documents  and  guidebooks  questioning  the  value  of  a  checklist’s  ability  to  capture  actual 
manufacturing  program  risk.  This  posture  implies  that  the  use  of  checklists  is  not 
adequate  to  the  task  of  assuring  process  capability.  If  so,  the  DAG  is  correct  in  calling  for 
a  knowledge-based  acquisition  approach  but  does  not  actually  integrate  this  process 
approach  into  best  practices  as  found  in  the  non-DOD  industries.  The  DAG  recognizes 
that  a  demonstration  of  process  capability  requires  the  use  of  statistical  control  from  an 
assessment  of  key  characteristics  but  the  DOD  approach  lacks  policy  or  standardization 
in  production  decisions  acquisition.  The  DOD’s  current  risk  assessment  guidance 
describing  manufacturing  readiness  is  therefore  favored  over  a  knowledge-based 
readiness  assessment  (USD[AT&L]  2013). 
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Examining  authorities  involved  in  the  assessment  and  approval  process  in  DOD 
acquisition  provides  a  potential  construct  for  understanding  the  current  state  and  any 
improvement  potential  in  determination  of  DOD  manufacturing  readiness  at  MS  C.  The 
PM  as  a  program  authority  is  assigned  the  responsibility  for  executing  a  SE  approach  in 
accordance  with  acquisition  policy  and  guidance.  The  PM  is  assisted  by  the  appointment 
of  a  technical  authority  (TA).  These  authorities  are  two  but  not  equal  authorities.  The 
manifestation  of  this  is  important  to  the  production-readiness  approach  found  in  the 
automotive  and  regulatory  industries.  The  non-DOD  industries  employ  the  use  of  a 
certified  production  readiness  document  called  a  Product  Submission  Warrant  (PSW) 
assuring  conformance  to  product  and  process  requirements.  The  certified  warrant  is 
signed  by  a  producer’s  production  authority  as  an  industry  best  practice.  The  DOD  does 
not  have  a  similar  certification  process  for  production  readiness  for  PD  start. 

If  the  DOD  TA  structure  supported  a  PSW  type  policy  it  would  first  require  an 
accepted  standard  for  production  readiness  demonstration.  A  white  paper  by  Ireland 
(2017)  provides  a  detailed  discussion  for  such  a  TA  approach.  Ireland’s  paper  discusses 
how  a  certificated  warrant  for  production  readiness  could  be  created  by  identifying  a 
production  readiness  demonstration  as  a  key  performance  parameter  (Appendix  E  -  DOD 
Technical  Warrants).  In  brief,  the  organizational  construct  of  a  Technical  Warrant  Holder 
derives  within  a  DOD  system  command  (SYSCOM)  workforce  grouping  the  authority  to 
manage  a  certification  process  to  demonstrated  production  readiness.  This  workforce 
grouping  represents  a  Competency-Aligned  Organization  that  holds  the  recognized  TA 
group.  The  certification  warrant  for  production  readiness  is  then  the  evidence  and 
confirmation  demonstrating  production  capability. 

Additional  supporting  justification  for  the  PSW  being  a  best  practice  can  be  found 
in  the  recently  released  Society  of  Automotive  Engineer  (SAE)  standard,  AS-9145 
Advanced  Product  Quality  Planning  (APQP)  and  Production  Part  Approval  Process 
(PPAP)  (SAE  2015).  The  warrant  process  provides  a  commitment  by  the  producer  that  a 
production  process  is  effective  and  suitable  at  production  start. 
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2.  Manufacturing  Development  and  Consensus  Standards 

Since  1996,  the  GAO  had  reported  poor  production  outcomes  and  related  this  to  a 
lack  of  a  demonstrated  process  capability.  At  the  time,  the  DOD  acquisition  system  of 
1996  eliminated  most  military  standards  due  to  the  post  Perry  memo  period.  Here,  the 
force  of  law  required  the  use  of  commercial  consensus  standards  (Breitenberg  1996)  and 
(Perry  1994).  Consider  the  discussion  taken  from  this  1996  GAO  report: 

In  December  1995,  DOD  began  the  Single  Process  Initiative,  managed  by: 

DCMC  that  allows  contractors  with  military  contracts  to  transition  their 
quality  management  system  from  MIL-Q  985 8 A  to  their  best  practice, 
such  as  a  quality  management  system  based  on  ISO  9000,  the  basic 
commercial  standard.  The  response  to  date  has  been  slow;  as  of  June  5, 

1996,  38  contractors  had  submitted  proposals  to  change  their  quality 
management  systems,  5  of  which  had  been  approved  (15). 

The  National  Institute  of  Standards  and  Technology/GAO  (1996)  report  stopped 
short  of  recommending  the  newly  commercialized  automotive  family  of  standards  over 
the  ISO-9000  family  of  standards.  The  ISO-9000  QMS  had  been  the  accepted  practice  for 
contractors  in  DOD  acquisition.  The  DOD  acceptance  of  the  aerospace  standard  (AS) 
AS-9100  was  not  a  significant  change  due  to  the  similarity  in  content  to  the  ISO-9000 
QMS.  The  adoption  of  these  QMS  replaced  the  use  of  military  standards  for  contractor 
quality  practices  that  prior  had  many  prescriptive  manufacturing  related  requirements. 

Given  the  two  types  of  QMS  (prescriptive  and  non-prescriptive),  one  finds  that 
the  difference  between  ISO-9000  and  AS-9100  is  minor,  but  the  difference  between  ISO- 
9000  and  QS-9000/ISO/TS- 16949  was  significant.  The  hallmark  difference  is  found  in 
the  prescriptive  methods  of  the  automotive  QS-9000  family  of  standards  and  guidebooks. 
A  key  tenant  of  the  QS-9000  QMS  was  the  application  of  a  PSW  or  certification  for 
product  and  process  conformance  verification.  The  PSW  requirement  included  customer 
specific  requirements  such  as  statistical  process  control  of  the  manufacturing  processes. 
Dimensional  requirements  would  be  satisfied  if  a  demonstration  to  a  specified  threshold 
performance  was  achieved  utilizing  the  indices  process  performance  index  (Ppk)  or 
process  capability  index  (Cpk).  Demonstration  of  process  capability  knowledge  at 
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production  decision  would  overcome  the  patchwork  of  checklists  that  rely  upon  the  risk- 
based  approach  of  the  DOD. 

A  significant  operational  difference  found  in  the  standard  automotive  OEM 
customer  quality  requirements  was  the  passing  of  their  common  requirements  on  to  lower 
tier  suppliers  as  an  enabling  factor  towards  the  success  of  the  automotive  sector.  The 
enhanced  set  of  automotive  quality  requirements  included  how-to  manuals  as  guidance 
for  APQP  with  a  PPAP  requirement  (Table  4).  As  manufacturing  processes  developed 
under  the  guidance  of  APQP  best  practices  the  expectation  of  a  supplier’s  validation  of 
product  and  production  readiness  completed  by  showing  empirically  any  supplier’s 
readiness. 

Table  4.  Automotive  Prescriptive  Guidance  and  PPAP  Quality  Standards 


Global  Automotive  Standards 

For  the  OEM  Supply  Network 

Source 

G-Guide 

R  -  Required 

ISO/TS  16949  -  Quality  Management  Systems 

Particular  requirements  for  the  application  of  ISO  9000:2008  for 
automotive  production  and  relevant  service  part  organizations 

ISO.org 

R 

Advanced  Product  Quality  Planning  and  Control  Plan  (APQP) 

The  APQP  and  Control  Plan  is  a  reference  manual  that  streamlines  the 
entire  quality  &  manufacturing  process  control  approach  in  support  of  a 
development  program.  Provides  a  means  to  communicate  requirements 
to  suppliers. 

AIAG.org 

G 

Design/Process  Failure  Mode  &  Effects  Analysis  (DFMEA/PFMEA) 

As  part  of  the  APQP  family  of  reference  manuals  this  guide  answers 
how-to  perform  a  DFMEA  and  PFMEA. 

AIAG.org 

G 

Measurement  Systems  Analysis  (MSA) 

This  guide  assists  in  the  assessment  of  a  measurement  system  that 
supports  a  manufacturing  process. 

AIAG.org 

G 

Statistical  Process  Control  (SPC) 

This  guide  provides  includes  a  wide  range  of  statistical  methods  for 
effective  monitoring  and  control  of  manufacturing  processes. 

AIAG.org 

G 

Production  Part  Approval  Process  (PPAP) 

The  supply  network  is  required  to  comply  with  the  requirements  of  the 
PPAP.  Consistent  quality  demonstrated  in  an  actual  production  run  at 
production  rates.  The  PPAP  integrates  production  readiness  including  the 
design,  qualification,  process  capability  with  a  certification  warrant. 

AIAG.org 

R 
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E.  PRODUCTION  AND  PROCESS  CAPABILITY  DISCUSSION 

In  order  to  place  into  context  the  specialized  understanding  of  production  and 
process  capability  there  is  a  need  to  draw  from  experience  and  research  to  develop  a  basic 
treatment  on  the  subject.  This  understanding  will  help  enable  the  reader  to  benefit  from 
this  study’s  assessment  method  of  critical  criteria  used  in  the  AOA.  This  is  of  concern  to 
this  research  because  of  the  lack  of  guidance  in  DOD  acquisition  when  contrasted  to  the 
more  disciplined  and  robust  manufacturing  development  practices  present  in  non-DOD 
organizations  (Schinasi  2002). 

This  section  discusses  a  context  of  production  and  process  capability  to  show 
linkages  to  some  of  the  manufacturing  causal  knowledge  gaps  that  lead  to  poor 
production  outcomes.  Several  definitions  have  been  created  to  give  additional  clarity  to 
the  discussion  on  manufacturing  environments  and  related  contractor  capabilities: 

•  Manufacturing  development  -  associates  the  development  of  a  production 
environment  to  be  used  in  the  fabrication  or  production  of  an  item. 

•  Production  capability  -  is  used  as  a  term  that  associates  the  totality  of  an 
organization’s  ability  to  produce  an  item:  man,  machine,  equipment, 
facilities  and  methods. 

•  Process  capability  -  is  a  term  used  that  is  specifically  related  to  the  ability 
of  a  given  process  to  satisfy  a  design  tolerance  in  a  stochastic  manner. 

A  discussion  on  production  capability  examines  manufacturing  development  as  a 
producer’s  ability  to  define,  demonstrate  and  offer  its  ability  to  manufacture  a  product 
under  a  given  operational  strategy  that  gives  that  organization  a  position  in  their  chosen 
industrial  market  place.  The  maturity  of  a  manufacturing  development  that  defines  the 
fabrication  of  a  given  product  is  related  to  the  manufacturing  planning  needed  to  process 
and  demonstrate  that  ability  to  produce  at  a  desired  production  rate.  Each  item  produced 
under  this  manufacturing  development  is  to  meet  that  item’s  design  requirements.  The 
evidence  that  a  manufacturing  process  is  production  capable  is  the  knowledge-based 
approach  used  in  non-DOD  organizations  to  show  readiness. 
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It  is  the  DOD  and  non-DOD  production  approval  processes  that  can  be  studied  for 
their  approach  to  measure  a  new  production  capability  and  underlying  process  capability. 
The  pre-production  approval  results  are  used  to  show  the  importance  and  impact  of 
production  capability  and  therefore  readiness  at  production  start. 

When  process  capability  knowledge  is  lacking  then  poor  outcomes  may  occur  as 
decisions  are  made  without  sufficient  evidence  of  production  or  process  capability.  When 
a  production  decision  is  primarily  based  on  a  functional  qualification,  as  in  DOD 
production  readiness  decisions,  there  is  a  manufacturing  knowledge  gap.  The  DOD 
approach  to  manufacturing  development  may  show  production  feasibility  from  early 
fabrication  experience  but  leave  decision  makers  uninformed  about  process  capability. 
Most  non-DOD  industries  favor  a  manufacturing  development  similar  to  the  approach  in 
the  APQP  guidance  standards  published  by  the  AIAG  culminating  in  a  demonstration  of 
their  production  system  (AIAG  2008a). 

1.  What  Is  Process  Capability? 

An  item’s  technical  performance  qualification  is  no  guarantee  that  the  developed 
manufacturing  process  used  to  fabricate  that  item  is  capable.  A  process  that  is  required  to 
manufacture  an  item  has  its  own  process-dependent  performance  measures  to  substantiate 
that  the  there  is  a  capable  process.  When  a  qualified  item’s  functional  performance  meets 
requirements  it  cannot  confirm  that  the  same  item  was  made  under  manufacturing  process 
that  is  stable  and  in  control.  For  instance,  an  individual  process  step  can  have  a  high- 
degree  of  part-to-part  variation  due  to  equipment,  operators,  instructions,  and 
measurements.  When  a  fabricated  part  is  shown  to  meet  requirements  inference  cannot  be 
made  as  to  its  underlying  production  environment’s  capability. 

From  a  designer’s  view,  there  is  a  technical  data  package  with  drawings  that 
seldom  are  inclusive  of  any  specific  description  regarding  how  that  item  is  fabricated. 
The  producibility  aspects  of  a  design  are  beyond  the  given  design  feature  specified. 
Consider  a  hole  specified  on  a  drawing  with  its  dimensions  and  tolerance  that  is  sufficient 
to  communicate  a  designer’s  intent.  However,  if  one  wanted  to  create  that  hole  they  could 
select  from  a  myriad  of  fabrication  methods  unrelated  to  the  drawing  definition. 
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Whether  drilling,  stamping  or  a  water  jet  fabrication  method  is  chosen  it  is  vitally 
important  to  assure  manufacturing  method  translates  into  a  capable  process.  For  example, 
water  pressure  could  vary  at  the  time  of  processing  or  the  water  jet’s  effective  size  may 
vary  with  age  so  resulting  features  may  be  too  large  or  off-center  -  all  of  which  impact 
precision  and  accuracy  to  the  design  intent.  Process  capability  studies  are  the  typical  way 
to  provide  confidence  that  a  given  fabrication  method  of  a  design  feature  would  be 
producible,  affordable  and  process  capable.  These  processes  should  be  fully  designed  and 
verified  by  MS  C  in  DOD  acquisition  as  they  are  in  non-DOD  counterparts  as  a  pre- 
production  demonstration  requirement. 

In  another  case,  manufacturing  planning  and  execution  for  any  process  step  under 
some  level  of  process  control  must  involve  suitable  tools  and  fixtures  that  meet  the 
quantity  and  quality  challenges.  Functional  features  that  are  difficult  to  measure  at  the 
point  of  fabrication  may  rely  upon  a  downstream  process  functional  test  or  an  upstream 
process  characteristic  validation. 

For  example,  a  fastener  is  required  to  reach  a  certain  torque  with  the  intent  to 
assure,  indirectly,  a  clamp  load  that  engineering  defines.  If  the  clamp  load  characteristic 
is  critical,  then  certain  in-process  features  may  be  critical  to  the  process  for  controlling 
fastened  torque.  A  process  action  may  define  a  critical  characteristic  for  fastener  position 
(Soft  start,  thread  engagement  and  angle)  during  torque  achievement  in-station.  This 
process  may  need  to  be  inspected  by  a  functional  downstream  test  using  a  wrench  that 
measures  breakaway  torque.  Indirect  measurement  studies  can  support  critical  product 
characteristics  defined  as  supporting  process  tolerances  in  fabrication.  Items  made  as 
manufacturing  representative  prototypes  or  early  production  may  not  have  adequate 
production  controls  applied  in  the  manufacturing  processes  in  DOD  acquisition  in  the 
EMD  phase.  The  relationship  of  functional  performance  and  part  fabrication  controls  is 
fundamental  in  achieving  production  and  process  capability.  This  is  why  product 
qualification  testing  cannot  substantiate  process  qualification  directly. 

The  need  to  identify  and  measure  critical  characteristics  in  the  process  under 
manufacturing  control  is  a  best  practice  in  non-DOD  production  organizations  as  pointed 
out  by  Sullivan  (2008).  Over  time,  process  capability  needs  periodic  assessment  to 
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demonstrate  process  stability,  homogeneity  and  measurable  achievement  of  product 
performance  at  a  production  demand  rate.  Statistical  Process  Control  (SPC)  performance 
can  show  a  stable  process  exists  demonstrating  that  parts  are  in  conformance  with  design 
tolerances  and  can  be  correlated  to  product  performance.  Statistical  Process  Control 
reveals  the  degree  of  process  capability  and  assures  part-to-part  variation  is  acceptable  to 
a  threshold  quality  level  and  may  show  an  as-built  item  meets  design  performance. 

2.  Process  Capability  Measurement 

In  any  manufacturing  development  process,  capability  determination  may  include 
the  identification  of  some  risk  and  uncertainty  related  to  late  design  changes  anticipated 
from  early  test  results  and  production  methods  used  in  qualification.  Assessing  a 
production  readiness  of  an  item  solely  upon  a  risk-based  PRR  technical  review  falls  short 
when  trying  to  establish  a  production  maturity  level.  This  approach  is  insufficient  in 
contrast  to  the  evidence  of  compliance  that  assesses  part-to-part  variation  in  a 
demonstration  of  process  capability. 

In  the  DOD  acquisition  guidance,  a  contractor’s  production  plans  may  be 
complete  but  not  realized  in  EMD  where  manufacturing  development  should  complete. 
The  DAS  allows  manufacturing  development  to  continue  during  LRIP.  This  can  be  up  to 
ten  percent  of  a  production  order  and  be  repeated.  Therefore,  production  capability  is  not 
known  and  is  a  risk  at  the  end  of  EMD.  This  would  be  unacceptable  and  woefully 
incomplete  in  most  non-DOD  industries  and  would  not  be  allowed  in  regulatory  industry 
environments.  This  lack  of  assessing  actual  production  knowledge  is  a  typical  omission 
in  the  government  readiness  assessments. 

Non-DOD  industries  follow  standard  formulas  to  measure  process  capability.  A 
manufacturing  process  can  be  addressed  by  the  use  of  Cpk  defined  in  the  AIAG’s  SPC 
reference  manual  (AIAG  2005).  These  capability  indices  are  to  verify  the  ability  of  a 
process  to  meet  design  tolerance  requirements  in  a  repeatable  manner  and  at  production 
rates.  These  indices  can  be  correlated  to  measures  of  non-conformance  such  as  defects 
per  thousand  produced.  The  probability  of  violating  design  tolerances  are  set  as  threshold 
indices  and  are  correlated  by  a  process  measure  of  variation. 
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Manufacturing  assessments  are  needed  as  on-going  measures  of  process 
capability.  The  use  of  SPC  under  a  given  process  control  can  signal  adverse  trends  with 
excessive  part-to-part  variation  that  would  lead  to  a  non-conformance  condition.  Active 
capability  assessments  allow  an  ability  to  prevent  non-conformances  by  in-station 
adjustments  preserving  process  standards.  When  part  tolerance  limits  are  established  and 
measurement  systems  are  suitable  then  this  approach  is  an  industrial  best  practice.  Yang 
discusses  the  approach  to  automation  control  characteristics  in  quality  planning  in 
computer-aided  design  and  relies  on  the  following  basic  steps  (Yang  2007,  30): 

Here  are  the  steps  to  follow  when  implementing  SPC: 

•  Take  periodic  samples  from  process 

•  Plot  sample  points  on  control  chart 

•  Determine  if  process  is  within  limits 

•  Prevent  quality  problems 

While  Yang’s  research  developed  a  systematic  approach  to  analyze  the  tolerance 
stack-up  for  complex  multi-spec  processes,  his  outline  of  action  is  central  to  using  a 
knowledge-based  QMS  to  assure  quality  targets  are  achieved.  Process  capability  viewed 
through  the  analytics  of  Cpk  and  Ppk  measures  are  related  to  on-going  process  capability 
verification.  The  use  of  quality  metrics,  such  as  Cpk  and  Ppk,  help  to  verify  process 
control  achievement  of  design  tolerances  and  avoids  adverse  product  effects  from  process 
variation.  These  measures  defined  and  validated  early  in  support  of  production  approval 
would  give  support  to  continuous  improvement  initiatives  in  LRIP  and  FRP  rather  than 
continuing  uncertain  manufacturing  development  capability.  This  would  pull  back  fleet 
readiness  risk  into  manufacturing  development  prior  to  MS  C. 

It  is  important  to  note  how  the  GAO  advocated  for  this  practice  of  calculating 
process  control  indices.  The  GAO  discussed  best  practices  that  included  measurement  of 
defect  expectation  to  avoid  adverse  findings  on  DOD  acquisition  of  weapon  systems.  The 
GAO  found  recently  that  DOD  acquisition  practices  still  did  not  confirm  production 
readiness  with  a  demonstration  as  a  best  practice  (Dodaro  2014): 
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To  assess  production  maturity,  we  asked  program  officials  to  identify  the 
number  of  critical  manufacturing  processes  and,  where  available,  to 
quantify  the  extent  of  statistical  control  achieved  for  those  processes  as  a 
part  of  our  data-collection  instrument.  In  most  cases,  we  did  not  verify  or 
validate  the  information  provided  by  the  program  office.  We  clarified  the 
number  of  critical  manufacturing  processes  and  the  percentage  of 
statistical  process  control  where  information  existed  that  raised  concerns. 

We  used  a  standard  called  the  process  capability  index,  a  process- 
performance  measurement  that  quantifies  how  closely  a  process  is  running 
to  its  specification  limits.  The  index  can  be  translated  into  an  expected 
product  defect  rate,  and  we  have  found  it  to  be  a  best  practice  (159). 

3.  Measurement  System  Analysis 

Dimensional  control  in  a  manufacturing  process  requires  an  understanding  of  the 
associated  measurement  systems  that  are  used  to  confirm  a  manufactured  item  meets  a 
given  design  feature’s  specified  engineering  tolerance.  Common  within  a  DOD 
contractor’s  production  facility  are  the  inclusion  of  test  and  measurement  devices  to  show 
fabrication  and  assembly  processes  are  satisfying  functional  design  requirements.  The 
process  capability  to  be  demonstrated  and  maintained  requires  a  suitable  calibrated  test 
and  measurement  system’s  precision  and  accuracy  —  meaning  that  the  gage  or  measuring 
device’s  error  is  typically  designed  around  a  10:1  ratio  with  the  error  of  measurement 
being  at  most  10%  of  the  tolerance  of  interest  as  a  best  practice  (AIAG  2010).  The  ability 
to  measure  and  replicate  a  manufacturing  process  part  after  part  says  more  about  a 
manufacturing  process  capability  than  a  qualification-only  approach.  Production  process 
capability  measures  demonstrate  the  confidence  or  likelihood  that  a  production  process 
will  sustain  its  ability  to  make  product  that  meets  product  functional  requirements. 

One  experience  in  a  measurement  study  considered  a  relationship  between  a 
measurable  design  feature  and  its  measuring  device.  That  measuring  device  was  a 
calibrated  tape  measure.  The  tape  measure  had  graduated  markings  of  measurement  every 
l/32nd  inch.  As  a  rule  of  thumb  the  error  of  measurement  of  a  trained  operator  has  an 
accuracy  of  Vi  the  marked  division.  When  the  process  called  for  a  measurement  of  l/32nd 
inch  tolerance  this  meant  every  part  would  have  to  measure  nominal  to  the  center  of  a 
design  tolerance  with  no  margin.  Here,  the  error  of  measurement  (+/-  l/64th  inch)  used 
the  entire  engineering  specified  design  tolerance.  The  measuring  device  was  not  suitable. 
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Stochastically  the  probability  of  being  nominal  is  zero  and  therefore  every  part  was  non- 
conforming. 

4.  Process  Definition 

In  addition  to  the  quantitative  aspects  of  a  process  capability,  there  is  also  the 
fidelity  of  the  manufacturing  development  definition  that  ultimately  defines  the 
production  capability  of  an  organization  including  enabling  structures  listed: 

•  manufacturing  plans 

•  work  instructions 

•  production  methods 

•  production  machines 

•  production  tooling 

•  test  equipment 

•  facilities 

These  items  are  all  a  part  of  the  production  capability  context  prior  to  one  part 
being  ready  to  build  under  the  challenge  of  production.  These  items  are  integral  to  the 
homogeneity  of  process  relied  upon  prior  to  understanding  if  a  functioning  part  is  from  a 
well-defined  and  well-behaved  production  process  under  process  control  (AIAG  2005). 

5.  Advanced  Product  Quality  Planning  and  Risk 

The  advanced  product  quality  planning  process  is  directly  related  to  the 
automotive  QMS  and  concerns  itself  with  the  development  of  product  and  production 
capability.  APQP  has  four  core  product  and  manufacturing  development  strategies  and 
are  referenced  in  Table  4.  The  APQP  guidance  is  standard  practice  in  most  non-DOD 
industries  and  includes  the  definition  manufacturing  knowledge  at  production  start  not 
found  in  DOD  practices.  For  example,  the  DOD  use  of  the  MRL  assessment  does  not 
identify  any  process  capability  requirements  and  gives  no  guidance  on  DOD  oversight 
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with  respect  to  demonstrating  critical  characteristics  from  an  actual  production  line  when 
completing  EMD  at  MS  C. 

On  a  scale  of  one  (not  ready)  to  ten  (experienced),  the  MRL  of  8  is  the  maturity 
level  desired  at  the  end  of  EMD  (OSD  2012).  MRL  practitioners  find  that  there  is  a 
limitation  in  the  use  of  the  MRL  preset  questions  when  interrogating  a  manufacturing 
process  as  to  its  maturity  level.  The  MRL  approach  is  not  necessarily  able  to  identify 
project-specific  risks. 

A  process  study  serves  to  identify  any  associated  risk  or  non-conformance 
potential  in  a  manufacturing  process  which  might  otherwise  go  undetected.  In  contrast, 
trying  to  document  manufacturing  risk,  from  the  interrogation  approach  of  the  MRL 
assessment,  lacks  the  specificity  to  address  a  given  production  or  process  capability.  The 
MRL  level  associated  with  a  production  capable  process  is  set  at  MRL  9  where  process 
controls  would  be  evaluated.  In  the  DOD  assessment  of  manufacturing  maturity  an  MRL 
of  9  is  not  required  until  prior  to  LRP.  In  contrast,  if  a  non-DOD  industry  would  apply  the 
MRL  process  it  would  have  an  MRL  goal  of  10  with  process  improvement  behaviors  at 
production  start.  See  “Appendix  M.  MRL  Levels  6  to  10”  and  (OSD  2015). 

A  non-DOD  manufacturing  development  best  practice  uses  a  process  study 
approach  to  assess  specific  manufacturing  risk  that  results  in  the  identification  of  a 
potential  need  for  a  process  control  as  a  mitigation  of  process  risk.  One  such  approach  is 
the  Process  Lailure  Modes  and  Effects  Analysis  (PLMEA)  identified  in  the  automotive 
APQP  (AIAG  2008b).  Analyzing  and  addressing  a  manufacturing  process  for  potential 
ways  a  production  process  might  fail  is  a  robust  study  into  good  production  process 
analysis  and  should  be  completed  prior  to  production  start  and  is  not  a  typical  tool  used  in 
DOD  acquisition. 

The  design  related  risk  assessment  is  found  in  the  application  of  the  Design 
Lailure  Modes  and  Effects  Analysis  (DLMEA)  and  the  PLMEA  is  for  the  process  failure 
mode  assessment  (AIAG  2008b,  AIAG  1992).  According  to  the  AIAG’s  LMEA  manual, 
the  DLMEA  is  conducted  ahead  of  the  PLMEA  to  identify  design  risk.  Design  risk  found 
in  a  DLMEA  can  potentially  be  mitigated  by  actions  going  back  to  design  (as  in  selecting 
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different  materials)  or  in  considering  a  manufacturing  control  if  design  tolerance  margin 
is  small  in  proportion  to  desired  process  capability. 

The  power  of  the  PFMEA  analysis  is  in  its  ability  to  identify  potential  ways  a 
specific  process  may  fail  and  then  to  capture  actionable  process  changes  that  may  assure 
a  part  processed  under  statistical  control  meets  requirements  (AIAG  2008b).  The  PFMEA 
is  a  preferred  non-DOD  best  practice  for  identifying  manufacturing  risk  and  utilizes  three 
risk  parameters.  Risk  assessed  in  this  way  identifies  potential  non-conformance  concerns 
by  severity,  occurrence  and  detection  risk  factors.  Here,  a  finding  of  risk  allows  an 
evaluation  for  acceptance,  mitigation  or  elimination  of  potential  failure  modes  at  the  root 
cause  level  early  in  manufacturing  development  towards  a  robust  manufacturing  process. 

Another  core  strategy  found  the  AIAG’s  APQP  is  the  use  of  SPC.  This  would  be 
the  in-process  measurements  that  are  to  be  tracked  and  stochastically  evaluated  for  trends 
that  might  threaten  achievement  of  a  characteristic  of  design  due  to  a  manufacturing 
process  or  methods  variability.  A  fourth  core  APQP  strategy  is  the  Measurement  System 
Analysis  (MSA)  process  that  evaluates  the  underlying  measurement  system  used  in 
manufacturing  whether  special  test  equipment  or  gages  and  tools.  A  key  tenet  of  any 
MSA  is  a  calibration  process  affirming  equipment  accuracy  is  maintained  to  assure  a 
process  measurement  tool  is  calibrated  to  known  standards.  In  non-DOD  industries  it  is  a 
best  practice  to  use  the  error  of  measurement  to  amend  process  acceptance  criteria  to  be 
more  restrictive  than  a  design  tolerance — a  guard  band.  This  adjustment  is  rarely  used  by 
contractors  in  DOD  programs  and  without  this  guidance  quality  defects  escape  the 
factory  unlike  non-DOD  industries  that  guard  band  measurement  error. 

F.  PRODUCTION  APPROVAL  PROCESSES 

Non-DOD  organizations  interviewed  by  the  GAO  had  industrial  sector  influences 
that  guided  organizational  use  of  specific  QMS  with  an  associated  production  approval 
practice.  Organizations  followed  a  product  development  strategy  according  to  their 
associated  OEM  or  related  regulatory  agency  with  their  unique  requirements.  An 
organization’s  industrial  sector  provided  the  larger  influence  that  guided  the  type  of  QMS 
adherence  (Table  5). 
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Applying  the  selection  filter  of  Table  5  there  are  three  alternate  production 
approval  processes  that  emerge  for  study  as  industrial  sectors.  These  alternatives  are 
grouped  into  the  industrial  sectors  governed  by  Federal  Aviation  Administration  (FAA), 
the  Food  and  Drug  Administration  (FDA)  and  the  automotive  industry.  These  three,  non- 
DOD  industrial  sectors  were  determined  to  be  suitable  alternatives  because  of  scale  and 
product  complexity  for  comparison  to  the  DOD  acquisition  practices. 


Table  5.  Identification  of  Alternate  Production  Approval  Processes 


Sources 

Company 

Sector 

Organizational 

Motivation 

Study 

Significance 

Within 

Study 

(Sullivan 

2008) 

Boeing 

Commercial 

Airplanes 

Aerospace 

Initial  payment  and  % 
withheld  until  final 
delivery  functions 
properly  AS  9 100 

AS9100 

FAR  Part  14 

Yes,  By 
Sector  - 
FAA 

(Sullivan 

2008) 

Intelsat 

Satellites 

Telecom 

High  Reliability 
favorable  terms  from 
the  underwriters 
incentivize  good  quality 
outcomes 

Mature 

Technologies 

No 

(Sullivan 

2008) 

American 

Airlines 

Aerospace 

incentivize  good  quality 
outcomes  AS9100 

AS9100 

FAR  Part  14 

Yes,  By 
Sector  - 
FAA 

(Sullivan 

2008) 

Siemens 

Medical 

Solutions 

Medical  Devices 

measures  process  yields 

FAR  Part  21 

Yes,  By 
Sector  - 
FDA 

(Sullivan 

2008) 

Kenworth 

Heavy 

Equipment 

requires  their  own 
investment 

Six  Sigma 

Supply  Chain 
Tier  1 

No 

(Sullivan 

2008) 

Cummins 

Heavy 

Equipment 

Poor  Quality  Motivated 
Large  Change  in 
Development  Process 
requires  their  own 
investment 

Six  Sigma 

Warranty 

No 
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Sources 

Company 

Sector 

Organizational 

Motivation 

Study 

Significance 

Within 

Study 

(Schinasi 

1996) 

Texas 
Instruments 
Defense 
Systems  & 
Electronics, 
Dallas 

Defense 

DOD  accepts  monetary 
risks  in  development 
(Often  pays  for  quality 
issues) 

Little  Supply 
Knowledge 

Yes,  By 
Sector  - 
DOD 

(Schinasi 

1996) 

Texas 

Instruments, 

Lubbock 

Electronics 

Semiconductor 

NA 

NA 

No 

(Schinasi 

1996) 

Delco 

Electronics 

Automotive 

(QS-9000,  APQP, 
PPAP) 

Supply  Chain 

Yes,  By 
Sector  - 
Auto 

(Schinasi 

1996) 

John  Deer 
Horicon  Works 

Lawn  &  Ground 
Care 

reduce  defects,  reduced 
suppliers,  reduced 
inspection 

Supply  Chain 

No 

(Schinasi 

1996) 

Varian 

Electronics 
Medical  Devices 

Malcorn  Baldrige  with 
advanced  quality 
systems 

Supply  Chain 

Yes,  By 
Sector  - 
FDA 

(Schinasi 

1996) 

Motorola 
Paging  Products 

Electronics 

Telecom 

Malcom  Baldrige  with 
advanced  quality 
systems 

Supply  Chain 

No 

(Schinasi 

1996) 

Cherry 

Electronics 

Products 

Automotive 

(QS-9000,  APQP, 
PPAP) 

Supply  Chain 

Yes,  By 
Sector  - 
Auto 

While  there  are  examples  of  DOD  contractors  who  applied  the  automotive  type 
QMS  to  their  supply  network,  it  is  rare.  United  Technologies  Corporation  (UTC)  is  one 
example.  The  history  of  UTC  included  an  automotive  division’s  influence  observed  by 
the  application  by  Sikorsky  (Appendix  L  -  Links  to  Quality  Practices). 

Table  6  illustrates  the  three  industrial  sectors  rough  scaling  of  business  metrics 
and  associated  financial  data.  A  gross  comparison  revealed  each  industry’s  economic 
statistics  in  terms  of  1)  number  of  employees;  and  2)  revenue  (Department  of  Commerce 
2015a;  Department  of  Commerce  2015b;  Ibisworld  2015).  The  scalable  comparison  of 
economic  factors  showed  that  each  industry  sector  had  definable  quality  systems.  The 
non-DOD  industries  studied  included  a  certification  process  as  part  of  their  production 
approval  process  not  observed  in  the  DOD  approach. 
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Table  6.  Rough  Scaling  of  Industry  Sectors 


Rough  Scaling  of  Industry 
(Selected  for  AOA  -  Production  Part  A 

sectors 

sproval  Processes) 

Sector 

Direct 

Billions 

Billions 

Employees 

Revenue 

Export 

Commercial  Aerospace 

500,000 

$216 

$118 

Automotive  Industry 

786,000 

$225 

$75 

Medical  Devices 

411,000 

$40 

DOD  -  three  largest  Contractors 

400,000 

$100 

With  the  assumption  that  better  production  outcomes  in  non-DOD  industries 
resulted  from  the  use  of  their  best  practices  the  reader  is  left  asking  what  differentiated 
these  processes  from  DOD  production  approval  practices?  A  review  of  these  non-DOD 
QMS  applications  found  that  they  all  used  a  more  prescriptive  type  of  QMS.  The  QMS 
groupings  provided  the  insight  needed  to  conduct  the  AOA  which  appears  in  detail  in 
chapter  III  of  this  report.  This  supports  why  a  quality  management  system’s 
characteristics  and  benefits  need  to  be  explored  in  this  study.  The  GAO  report  (2010) 
demonstrates  the  direction  the  DOD  selected. 

The  DOD’s  response  to  the  GAO  report  of  2010,  found  in  “Appendix  G,”  did  not 
satisfy  the  GAO’s  hopes  that  a  consistent  or  standard  approach  might  emerge  in  DOD 
production  approval  with  a  demonstration,  including  statistical  process  control  (SPC) 
(Sullivan  2010).  Capturing  the  key  response  from  the  DOD  memo: 

While  the  Department  [DOD]  notes  that  all  manufacturing  processes  do 
not  warrant  the  same  level  of  process  capability  and  control,  appropriate 
levels  of  control  are  certainly  warranted  on  a  case  by  case  basis  (66). 

and 

The  Department  [DOD]  will  examine  strengthening  the  manufacturing 
readiness  criteria  related  to  process  capability  and  control  of  critical 
components  and/or  interfaces  prior  to  the  MS  C  low  rate  initial  production 
decision.  However,  program  offices  and  contractors  should  continue  to 
have  the  latitude  to  jointly  agree  on  the  targets  and  specific  process  control 
demonstrations  required  on  the  pilot  production  line  during  the 
Engineering  and  Manufacturing  Development  to  ensure  success  (66). 
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G. 


PRODUCTION  APPROVAL  PROCESS  ATTRIBUTES 


Given  the  comparative  groupings  of  the  FAA,  FDA  and  the  auto  industry,  the 
next  goal  of  the  study  focused  on  the  development  of  an  analysis  tool  to  differentiate 
between  alternate  production  approval  processes.  A  review  of  each  industrial  sector’s 
approach  to  production  approval  practices  provided  a  producer’s  respective 
manufacturing  development  knowledge  and  attributes  of  each  production  readiness 
decision  process. 

The  comparative  study  criteria  developed  performance  and  cost  benefits.  A 
comparison  of  each  industrial  sector,  selected  for  study,  followed  a  discoverable  and 
formal  production  approval  process.  The  comparison  to  the  DOD  production  readiness 
process  used  high-level  criteria  developed  from  the  industrial  best  practices.  A  summary 
categorizing  production  process  characteristics  as  best  practice  attributes  are  presented  in 
Table  7  as  observed  in  the  researcher’s  commercial  experience. 

Each  statement  found  in  Table  7  is  a  best  practice  attribute  that  expresses  a  robust 
matured  practice.  For  example,  there  is  a  considerable  amount  of  professional  effort  in 
developing  consensus  standards  and  then  to  deploy  them  throughout  an  entire  industrial 
sector  supply  network.  It  is  not  unusual  for  a  commercial  standard  to  take  several  years  to 
gain  consensus  and  then  another  decade  to  maintain  and  discover  its  effectiveness. 


Table  7.  Input  and  Output  Best  Practice  Attributes 


Production  Process  Input  /  Output  Best  Practice  Attributes 

=3 

Oh 

1  G 

1 

Common  engineering  language 

2 

Common  quality  standards 

3 

Common  customer  requirements 

4 

Third  party  compliance  to  quality  system  standards 

5 

Prescriptive  advanced  quality  planning  practices 

6 

Quality  requirements  flowed  to  the  entire  supply  network 

7 

Certification  of  demonstrated  capability  prior  to  production 

Output 

1 

Quality  system  administrative  efficiencies 

2 

Knowledge-based  demonstration  of  manufacturing  capability 

3 

Measurable  system  of  quality  metrics  for  user  satisfaction 

4 

Improved  product  reliability  and  durability 
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H.  QUALITY  MANAGEMENT  SYSTEMS  (QMS) 

Quality  management  system  standards  were  found  to  influence  an  organization’s 
effectiveness  as  a  manufacturer.  Basic  production  capability  depended  upon  an 
organizational  approach  to  production  approval.  The  two  basic  types  of  QMS 
applications  included  the  ISO-9000  family  of  standards  and  the  Automotive  Industry 
Action  Group  (AIAG)  QMS  QS-9000  family  of  standards.  Depending  on  which  QMS  an 
organization  followed,  there  were  different  quality  outcomes  observed  (ISO  2008;  AIAG 
2006). 

The  ISO-9000  related  standards  remain  the  core  of  all  QMS  in  the  study.  A  QMS 
serves  as  an  industry  best  practice  employed  by  organizations  as  an  assurance  to 
customers  that  their  systems  and  processes  followed  good  management  principles  and 
quality  practices  that  influence  product  and  service  results.  The  QS-9000’s  release 
followed  closely  after  the  publication  of  the  ISO-9000  standards  but  added  customer 
specific  guidance  documents  requiring  prescriptive  requirements  directed  at  the 
automotive  industry  OEM  supply  chain. 

The  QS-9000  family  of  standards  unified  a  myriad  of  automotive  OEM  standards 
used  by  the  entire  automotive  supply  base  (Bandyopadhyay  1996,  7,  12).  Bandyopadhyay 
observed  that  registering  compliance  to  the  earlier  released  ISO-9000  family  of  standards 
was  not  satisfactory  for  the  automotive  OEMs  and  the  additional  requirements  of  the  auto 
industry  that  contributed  to  higher  quality  production  outcomes. 

The  AIAG  consortium,  formed  by  the  American  automotive  OEMs,  enabled  and 
managed  the  newer  enhanced  quality  standards  of  the  QS-9000.  As  reported  by 
Bandyopadhyay,  the  supply  network  implementation  of  the  unified  OEM  automotive 
approach  was  facilitated  by  the  AIAG  assisting  in  educating,  enabling  and  guiding 
suppliers  as  to  the  necessary  means  to  have  a  common  production  approval  process. 

The  supply  base  QMS  implementation  relied  upon  AIAG  training  and  third  party 
compliance  registration.  The  instrumental  role  of  the  AIAG  and  the  formulation  of 
common  standards  in  the  United  States  have  now  grown  to  include  most  automotive 
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OEMs  globally  and  are  practiced  by  the  entire  supply  chain  and  avoiding  multiple  QMS 
requirements.  See  Appendix  F.. 

A  review  of  the  automotive-unique  requirements  distinguished  the  two  QMS 
standards  (ISO-9000  and  QS-9000)  as  materially  different.  The  automotive  enhanced 
quality  standards  (QS-9000)  required  many  prescriptive  OEM  quality  practices  and  how 
to  conduct  a  demonstration.  Process  capability  required  the  identification  of  critical 
characteristics  during  their  disciplined  process  development.  The  additional  requirements 
were  essential  to  confirm  production  readiness  through  a  production  capability 
demonstration  prior  to  a  production  decision.  The  automotive  QS-9000  standards 
transferred  control  from  the  AIAG  to  the  SAE  and  preserved  the  automotive  OEM 
customer  specific  requirements  to  reflect  the  global  supply  base  managed  by  ISO.  This 
was  done  in  collaboration  with  an  international  automotive  taskforce  (AIAG  2006).  The 
ISO  rebranding  of  the  QS-9000  QMS  used  the  TS  prefix  designation  now  rendered  as 
ISO/TS- 16949.  This  allowed  the  ISO  construct  to  include  detailed  customer  specific 
requirements  of  a  prescriptive  nature. 

After  a  decade  of  use,  the  ISO-9000  QMS  type  was  assessed  by  a  survey 
conducted  by  McGraw-Hill  in  1999  (Naveh,  Marcus  and  Koo  Moon  1999).  Similarly,  for 
the  QS-9000  QMS  type  there  were  two  similar  surveys  by  the  AIAG  and  the  American 
Society  of  Quality  (ASQ)  (AIAG/ASQ  1997)  and  (AIAG  1998).  The  ISO-9000 
standards,  absent  of  prescriptive  practices,  served  as  a  management  system  of  quality 
related  policy  statements  that  identified  manufacturing  best  practices.  The  DOD  version 
of  ISO-9000  was  the  AS-9000  QMS.  The  more  prescriptive  QS-9000  standards  with  a 
certification  of  production  demonstration  represented  the  evidentiary  difference  between 
standards  (SAE  2009).  Discussion  on  the  results  of  the  ASQ/AIAG  QS-9000  QMS  and 
McGraw-Hill  ISO-9000  QMS  application  surveys  are  in  the  AOA  section,  IV. B. 

There  was  a  finding  of  ineffectiveness  showing  little  improvement  when  adopting 
the  QMS  from  the  ISO-9000  series.  In  contrast,  the  QS-9000  series  of  prescriptive 
standards  showed  significant  improvement  in  a  producer’s  business  and  quality.  The 
difference  found  between  QMS  types  was  pivotal  in  providing  insight  and  reliability  in 
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the  study  findings.  The  QMS  type  used  and  its  impact  on  quality  were  highly  correlated 
in  each  survey  due  to  the  numerous  respondents. 

The  application  of  a  QMS  and  how  oversight  is  applied  in  DOD  weapon  systems 
acquisition  follows  policy  level  guidance  where  the  various  SYSCOMs  flow 
requirements  to  acquisition  teams.  Consider  the  AS  version  of  the  ISO  9000  QMS  (AS- 
9100)  (SAE  2009).  The  Secretary  of  the  Navy  (SECNAV)  endorsed  the  AS-9100  QMS  in 
SECNAVINST  5000.2D/E  (“Appendix  D”). 

The  DOD  acquisition  leads  can  select  higher  level  quality  standards  such  as  a 
QMS  for  compliance  to  a  contractor’s  conduct  of  an  engineering  development  and 
execution  of  a  production  system.  Acquisition  teams  have  some  guidance  as  to  which 
consensus  standards  can  be  levied  against  a  contract.  The  usage  of  a  particular  QMS, 
such  as  the  ISO/TS- 16949  reference  within  the  CFR,  would  be  to  provide  a  more 
prescriptive  set  of  requirements  to  weapons  contracting.  The  GAO  report  (1996), 
discussed  the  use  of  the  ISO/TS- 16949  quality  standard  as  a  supplement  to  ISO-9000  but 
no  finding  of  its  use  has  been  shown  in  subsequent  GAO  reports  (Schinasi  1996,  13). 

The  CFR  called  out  the  need  for  each  agency  desiring  to  specify  a  higher  quality 
standard  to  create  a  procedure  to  assess  when  such  a  need  would  exist.  Unfortunately,  the 
DOD  acquisition  legacy  and  policies  only  guide  program  teams  to  follow  performance- 
based  acquisition  favoring  a  contractor’s  individual  approach.  Additionally,  DOD 
acquisition  policy  and  guidance  does  not  indicate  when  to  use  the  enhanced  QMS  like 
ISO/TS-16949  or  the  AS-9100  D  release  that  will  likely  be  required  after  2018  that  also 
includes  the  AS-9145  APQP  and  PPAP  manufacturing  development  and  quality 
production  approval  standard  in  the  overall  QMS  family. 

While  it  is  not  found  in  any  guidance,  the  current  AS-9100  revision  C  QMS  does 
allow  for  a  customer’s  special  or  discretionary  requirements  (Table  8).  Invoking  the 
special  requirements  clause  allows  regulatory  requirements  to  be  defined  to  satisfy 
supplementary  agency  requirements  to  be  levied  by  the  FAA  and  the  FDA.  The  DOD 
does  not  engage  on  this  level  of  requirements  as  a  standard  practice  or  even  selectively. 
DOD  Prime  contractors  act  as  the  OEMs  in  acquisition  with  the  DOD  being  the  end  user. 
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Non-DOD  supplier  contracting  relationships  develop  differently  and  more  directly 
defining  the  relation  between  a  prime  and  a  subcontractor  (SAE  2009).  A  summary  of 
AS-9100,  clause  3.2,  given  by  the  Long  Island  chapter  of  ASQ  has  discretionary 
language  almost  identical  to  that  shown  in  the  instruction  given  by  the  Navy  policy 
guidelines  but  still  leans  toward  contractor  discretion  (ASQLONGISLAND  2015). 


Table  8.  AS-9100  C:  2008  Clause  3.2  -  Special  Requirements 


3.2  Special  Requirements 

Insight 

Those  requirements  identified  by  the  customer,  or 
determined  by  the  organization,  which  have  high  risks  to 
being  achieved,  thus  requiring  their  inclusion  in  the  risk 
management  process.  Factors  used  in  the  determination  of 
special  requirements  include  product  or  process  complexity, 
past  experience  and  product  or  process  maturity.  Examples 
of  special  requirements  include  performance  requirements 
imposed  by  the  customer  that  are  at  the  limit  of  the 
industry’s  capability,  or  requirements  determined  by  the 
organization  to  be  at  the  limit  of  its  technical  or  process 
capabilities. 

Why: 

Ensure  that  such  requirements  are  systematically 
addressed  and  linked  to  risk  management  activities  by 
the  organization 

Impact: 

A  formal  approach  to  identifying  special  requirements 
and  connecting  them  to  the  risk  management  process 

Adapted  from  American  Society  of  Quality  -  Long  Island  Division,  (2015). 


I.  LITERATURE  REVIEW 

Reviews  of  certain  DOD  policy  documents,  academic  publications,  technical 
publications  and  oversight  reports  help  identify  the  current  state  of  DOD  acquisition 
practice.  This  information  confirms  the  manufacturing  knowledge  gaps  identified  by  the 
GAO  as  deficiencies  and  causal  factors  for  poor  production  outcomes.  This  literature 
review  provides  sufficient  detail  with  respect  to  establishing  the  alternate  process 
knowledge  to  build  the  development  of  the  AOA  approach.  Observations  concerning 
alternative  production  approval  processes  established  a  solid  foundation  for  process 
comparison  in  the  AOA.  Collectively,  this  information  provided  the  development  of  the 
study  focus  and  the  process  performance  factors  for  the  AOA  assessment. 

There  are  three  factors  of  poor  production  outcomes  assessed  in  the  literature 
review  and  are  taken  from  Figure  1  identifying  the  reason  for  the  study  focus  on  lack  or 
manufacturing  knowledge: 
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1. 


Lack  of  Technical  Readiness 


a.  Technical  Readiness  in  the  TMRR  Phase 

Technology  knowledge  gaps  and  the  value  of  technology  readiness  in  DOD 
acquisition  are  explored  in  a  thesis  by  Coble  et  al.  (2014).  This  work  addressed  a  need  to 
identify  a  better  modeling  approach  in  requirements  definition  during  the  Technical 
Maturity  and  Risk  Reduction  (TMRR)  phase.  The  thesis  study  team  reported  an  improved 
requirements  modeling  approach  related  to  prototyping.  The  authors  believed  that 
improved  modeling  of  requirements  and  the  prototyping  process  would  help  more 
accurate  and  timely  facilitation  of  technology. 

Here,  Coble  et  al.’s  report  failed  to  identify  manufacturing  technology  knowledge 
gaps  as  being  of  any  concern  during  the  TMRR  phase.  The  work  by  Coble’s  team  is  part 
of  a  large  body  of  published  work  focusing  on  technological  maturity  in  DOD 
acquisition.  The  omission  of  manufacturing  development  in  Coble’s  team  thesis  is  similar 
to  other  research  ignoring  manufacturing  causality  on  poor  production  outcomes. 

b.  A  Broad  Meta  Study  Review  on  Technical  Readiness 

The  technical  report  by  Azizian,  Sarkani  and  Mazzuchi  (2009),  “A 
Comprehensive  Review  and  Analysis  of  Maturity  Assessment  Approaches  for  Improved 
Decision  Support  to  Achieve  Efficient  Defense  Acquisition,”  discussed  the  need  to 
address  poor  production  outcomes  of  major  weapon  system  programs.  Azizian,  Sarkani 
and  Mazzuchi’s  article  cited  many  of  the  same  GAO  reports  referenced  in  this  study  with 
the  same  causal  issues  for  poor  production  outcomes.  They  specifically  mention  all  three 
of  the  GAO  causal  factors  (including  manufacturing  knowledge  gaps)  but  only  reported 
on  issues  related  to  short  comings  of  technological  maturity.  This  team  joins  other 
researchers  omitting  the  manufacturing  causal  factor. 

According  to  Azizian,  Sarkani  and  Mazzuchi,  there  exists  confusion  in  the 
literature  with  respect  to  saying  a  technology  is  mature  or  if  a  technology  possesses  a 
readiness  to  the  extent  that  they  use  the  terms  interchangeably.  This  researcher  does  not 
agree  with  the  Azizian  team’s  interchangeability  of  terms.  In  studying  production 
approval  there  needs  to  be  clarity  between  maturity  and  readiness.  Maturation  is  to  be 
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viewed  as  a  process  and  readiness  is  the  assessment  of  maturity  at  a  point  in  time  against 
a  goal  or  standard.  The  confusion  can  be  resolved  by  the  context  given  in  the  respective 
Technical  Readiness  Level  (TRL)  and  MRL  guidebooks.  A  given  “readiness”  level  is  the 
result  of  using  the  related  maturity  assessment  process  asking  progressive  questions 
interrogating  a  product’s  performance  or  a  production  environment’s  definition. 

c.  Technical  Readiness  Contrasted  with  Manufacturing  Readiness 

A  review  of  the  DOD  acquisition  policy  and  guidance  documents  identified  the 
DOD  instruction,  DODI  5000.02,  which  describes  detailed  acquisition  policy 
requirements  (USD[AT&L]  2015).  These  DOD  policy  documents  deal  objectively  with 
technology  readiness  in  acquisition.  The  JCIDS  manual  states  that  for  any  critical 
technology  element  identified,  the  technical  maturity  measurement  must  satisfy  a  TRL  6 
prior  to  entry  to  the  EMD  phase  (JCS  2015b).  The  JCIDS  manual  shows  that  technology 
maturation  is  to  be  determined  by  a  demonstration  of  a  TRL  requirement  (JCS  2015b); 
(USD[AT&L]  2015). 

There  is  no  complementary  MRL  readiness  level  required  as  a  milestone  entrance 
or  exit  requirement.  As  a  measure  of  risk,  the  MRL  would  be  reported  to  the  MDA  only 
if  the  PM  considered  that  risk  significant.  The  absence  of  a  complimentary  manufacturing 
readiness  requirement  demonstration  inhibits  knowing  if  a  program  is  ready  to  proceed 
from  EMD  to  the  PD  phase.  A  production  environment  is  the  focus  of  the  desired  MRL 
level  and  is  not  based  on  a  production  environment’s  execution  under  production  control. 
The  MRL  guidebook  actually  de-emphasizes  the  MRL  “number”  in  favor  of  a  maturity 
assessment  and  any  plan  to  mature  any  discovery  of  shortcomings  (OSD  2012). 
Therefore,  there  is  a  lack  of  manufacturing  readiness  knowledge  by  DOD  guidance. 

2.  Lack  of  Design  Stability 

The  DOD  acquisition  policy  reformers  released  DOD  5000.02  in  2008  wherein  a 
formal  Preliminary  Design  Review  (PDR)  occurs  prior  to  Milestone  B  at  the  end  of  the 
TMRR  phase.  See  Ligure  2  (USD[AT&L]  2008).  The  DOD  acquisition  process,  as 
amended  in  2008,  inserted  the  PDR  technical  review  into  the  earlier  TMRR  phase  instead 
of  at  the  start  of  the  EMD  phase. 
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Therefore,  a  design  definition  in  EMD  is  now  definable  in  TMRR  to  assure  a 
more  stable  PDR  product  definition  in  the  build-to  configuration  with  its  allocated 
baseline  realized  prior  to  EMD.  At  this  point,  a  contractor  will  specify  a  design  level  that 
defines  the  PDR’s  allocated  baseline  and  apply  configuration  management  for  the 
associated  item(s)  developed.  Having  configuration  management  and  a  design  definition 
from  a  successful  PDR  in  the  TMRR  phase  is  discussed  as  addressing  the  GAO  concern 
related  to  design-instability  by  the  Office  of  the  DODR&E,  Systems  Engineering 
Department  (Dahmann  and  Kelley  2009). 

While  the  Dahmann  and  Kelley  paper  offered  guidance  on  the  approach  used  by 
system  engineers  during  development  phases  the  guidance  offered  failed  to  establish  an 
approach  with  respect  to  manufacturing  even  though  the  manufacturing  competency 
belongs  within  the  domain  of  SE.  There  is  no  mention  given  to  the  establishment  of 
manufacturing  requirements  or  capabilities  in  the  early  phases  leading  to  EMD  or  exiting 
from  EMD  to  enter  PD  start. 

There  is  ample  evidence  in  the  published  literature  and  the  current  DAS  showing 
that  the  two  causal  factors  identified  by  the  GAO  for  poor  production  outcomes  related  to 
technological  readiness  and  design-instability  have  been  addressed.  Therefore,  current 
acquisition  policy,  in  general,  closed  two  of  the  three  deficiencies  advocated  by  the  GAO 
reports  as  related  to  the  causality  of  poor  production  outcomes  after  MS  C.  On  the  other 
hand,  the  DOD  acquisition  reformers  have  not  shown  improvement  over  the  prior 
policies  with  respect  to  manufacturing  in  DOD  acquisition  (JCS  2008);  (USD[  AT&L] 
2013);  (JCS  2015a);  (JCS  2015b)  and  (USD[AT&L]  2015). 

3.  Lack  of  Manufacturing  Knowledge 

This  section  examines  various  selections  of  the  DAS  guidance  and  provides  a 
detailed  look  at  the  treatment  of  the  production  approval  processes  in  DOD  acquisition. 
One  key  observation  is  that  there  are  no  standards  that  require  a  demonstration  of 
statistical  control  of  a  manufacturing  process  prior  to  MS  C.  Consider  the  DOD 
acquisition  manufacturing  policy  guidance  in  the  JCIDS  manual  that  states  that  the  CPD 
should  be  “informed”  with  a  manufacturing  readiness  assessment  requirement  completed 
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in  accordance  with  the  Joint  Service/Industry  MRL  Working  Group’s  published  MRL 
Deskbook  (OSD  2012).  The  review  of  the  2015  JCIDS  manual  also  identifies  the  need 
for  manufacturing  knowledge  to  support  a  production  decision.  In  addition,  it  should  be 
noted  that  the  MRL  Deskbook  identifies  itself  as  a  best  practice  and  not  a  DOD 
requirement  in  conflict  with  the  5000.02  instruction  calling  for  its  use  in  weapons 
acquisition  as  a  requirement  (OSD  2012,  i)  and  (OSD  2015,  i). 

The  discussion  that  follows  considers  selections  of  the  DAS  taken  from  certain 
policy  documents  with  the  relevant  part  to  this  thesis  giving  a  brief  assessment: 

a.  Reference  Appendix  E  Selections  from  DOD  Acquisition  Policy:  Item  4 
Selection  1,  Selection  2  and  Selection  3 

Guidance  from  the  DODI  5000.02  includes  a  call  to  have  an  effective 
demonstration  of  production  capability.  However,  when  more  detail  in  this  requirement  is 
presented  in  the  instructions  there  is  a  demotion  from  a  rigor  associated  with  a 
demonstration  to  a  discussion  on  risk  to  support  CPD  reporting  requirements  to  satisfy  an 
MDA.  These  passages  advocate  for  a  manufacturing  demonstration  at  the  end  of  EMD 
and  also  prior  to  FRP.  The  need  for  a  manufacturing  demonstration  at  the  end  of  EMD  is 
subsequently  deconstructed  from  a  knowledge-based  demonstration  of  process  control  to 
a  risk  assessment  process  not  requiring  actual  evidence  of  production  process  capability. 

b.  Reference  Appendix  E  Selections  from  DOD  Acquisition  Policy:  Item  4 
Selection  4 

When  contrasting  the  manufacturing  demonstration  and  product  qualification 
required  for  reporting  technological  and  performance  requirements  at  the  end  of  EMD 
one  finds  that  the  manufacturing  demonstration  does  not  occur  and  is  deferred  until  after 
LRIP  instead  of  being  verified  as  an  exit  requirement  for  EMD.  Functionally  defined 
technical  requirements  appear  in  various  detailed  programmatic  documents  such  as  the 
TEMP,  statements  of  work  or  specification  documents  and  lists  of  performance  based 
requirements  to  be  demonstrated.  These  functional  requirements  for  demonstration  and 
qualification  prior  to  production  decision  are  only  product  based  and  not  process  based. 
Any  manufacturing  readiness  is  effectively  a  paper  study  with  readiness  being  reported  as 
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risk  to  the  contractor’s  manufacturing  plan.  Any  expectation  for  a  production  capability 
demonstration  does  not  factor  into  DOD  acquisition  at  MS  C. 

The  character  “M”  found  in  the  abbreviation  EMD  is  to  represent  a  program’s 
manufacturing  development  as  a  partner  of  design.  When  comparing  non-DOD 
development  the  M  is  properly  represented  with  respect  to  assessing  manufacturing 
design  and  development  with  a  verification  and  demonstration.  DOD  acquisition  waits  to 
assess  production  capability  as  a  post-production  consideration  at  the  PCA  event 
(USD[AT&L]  2015,  29). 

One  rationale  behind  the  omission  of  a  manufacturing  demonstration  in  DOD 
acquisition  is  by  mistakenly  relying  upon  functional  product  qualification  to  substitute 
product  performance  and  manufacturing  capability.  In  a  non-DOD  production  approval 
demonstration  there  is  a  requirement  to  show  verification  from  a  controlled  and  defined 
production  environment,  at  a  production  rate,  from  production  tooling,  with  product  built 
under  SPC  (AIAG  2005). 

In  addition,  the  DOD  policy  documents  do  not  engage  the  supplier  network  to 
assure  that  the  government’s  interest  in  being  production  ready  at  MS  C.  In  this  case,  a 
supplier’s  part  only  requires  a  functional  qualification.  The  flow  down  requirement  to 
show  that  manufacturing  processes  will  be  under  statistical  control  in  the  supply  base  is 
significant  and  different  than  accepting  a  part’s  qualification  results  as  representing  the 
capability  of  the  manufacturing  process.  Therefore,  any  program  that  fails  to  secure 
production  capability  knowledge  at  production  commitment  has  a  built-in  manufacturing 
knowledge  gap  that  is  a  causal  condition  for  poor  production  outcomes. 

The  DAS  higher-level  guidance  flows  down  in  DOD  acquisition  definition  to  the 
SYSCOMs.  Individually,  SYSCOMs  may  structure  a  responsive  instruction  to  achieve 
production  readiness  knowledge.  One  SYSCOM  instruction  directed  attention  to  a 
functional  performance  demonstration  and  production  process  readiness  and  planning 
asking  for  “a  satisfactory  basis”  for  determining  readiness  as  found  in  the  Naval  Air 
System  Command  Instruction  (NAVAIRINST)  in  NAVAIRINST  4355. 19D,  (NAVAIR 
2008) 
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c.  Reference  Appendix  E  Selections  from  DOD  Acquisition  Policy:  Item  4 
Selection  5 

In  the  current  NAVAIRINST  4355. 19E  there  is  far  less  guidance  given  with 
respect  to  manufacturing  readiness  and  removes  any  notion  of  an  objective  basis  for 
proceeding  to  LRIP  (NAVAIR  2015).  Clearly,  the  writers  of  the  D  revision  anticipated 
some  basis  of  satisfaction  to  be  determined  but  did  not  clearly  define  the  basis  or  require 
an  objective  demonstration  or  certification  of  production  readiness  as  recommended  by 
the  NDIA  in  2010.  Between  revisions  of  the  instruction  from  D  to  E,  there  is  a  scaled 
down  PRR  criteria  offered  by  the  SYSCOM  instruction  NAVAIRINST  4355. 19E 
(NAVAIR  2015). 

d.  Reference  Appendix  E  Selections  from  DOD  Acquisition  Policy:  Item  4 
Selection  6 

Here,  the  E  revision  referring  to  the  PRR  technical  review  process,  establishes 
less  content  rich  manufacturing  assessment  criteria  through  the  Critical  Design  Review 
(CDR)  and  as  represented  in  production  readiness  assessments  in  keeping  with  the 
reporting  of  risk  approach  in  the  higher  guidance  of  the  JCIDS.  The  conclusion  from  this 
literature  review  is  to  recognize  that  a  contractor  establishes  its  own  manufacturing 
requirements  separate  from  DOD  oversight  and  verifies  compliance  to  produce  an  item 
by  product  functional  qualification.  The  acquisition  guidance  then  assumes  the  risk  of  a 
contractor’s  production  capability  at  MS  C.  The  manufacturing  process  knowledge  that 
may  be  concurrently  developed  in  EMD  is  lacking  objectivity  and  may  be  misleading  or 
incomplete  from  the  PRR  conducted  in  support  of  MS  C. 

Decision  authorities  lack  an  empirical  conformance  and  demonstration  of 
production  capability  in  the  DOD.  Even  the  GAO  manufacturing  knowledge  score  card 
approach  lacked  an  actual  demonstration  giving  credit  in  favor  of  the  contractor  to 
proceed  to  LRIP  on  a  promise  that  manufacturing  will  be  under  statistical  control  at  some 
future  time  -  effectively  an  unsecured  I-Owe-You  to  exit  the  EMD  phase  (Dodaro  2013). 

This  is  similar  to  that  reported  by  the  Congressional  Research  Service’s  report 
stating  “At  Milestone  C,  the  MDA  authorizes  the  beginning  of  low-rate  initial  production, 
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which  is  intended  to  prepare  manufacturing  and  quality  control  processes  for  a  higher  rate 
of  production  and  provide  test  models  for  operational  test  and  evaluation  (OT&E).  Upon 
completion  of  OT&E,  demonstration  of  adequate  control  over  manufacturing  processes, 
and  with  the  approval  of  the  MDA,  a  program  can  go  into  full  rate  production”  (Schwarz 
2013,  10).  Therefore,  actual  manufacturing  knowledge  is  not  required  in  establishing  a 
developed  manufacturing  capability  at  MS  C. 

The  DAS  never  discusses  directly  that  product  qualification  is  a  substitute  for  a 
manufacturing  process  capability  demonstration  but  many  program  SOWs  have  stated 
this  in  the  author’s  experience.  The  manufacturing  verification  that  eventually  is  assessed 
is  deferred  until  the  PC  A  and  still  fails  to  gain  insight  to  supplier  parts.  Consider  further 
the  significance  that  the  supply  network  is  over  65%  of  content  purchased  or  developed 
(Sullivan  2010).  This  means  that  manufacturing  process  knowledge  is  nearly  invisible  to 
DOD  oversight  in  the  sub-tier  supply  network.  Beyond  view  are  quality  factors  that 
indicate  production  capability  such  as,  scrap  rates,  process  variability  and  first-pass-yield. 
As  long  as  deliveries  are  met  and  a  part  had  been  qualified  then  no  additional  supplier 
insight  is  required  -  including  any  production  capability. 

e.  Reference  Appendix  E  Selections  from  DOD  Acquisition  Policy:  Item  4 
Selection  7 

The  TRL  process  and  technical  risk  of  critical  elements  is  considered  a  rigorous 
process  in  the  TMRR  acquisition  phase  that  identifies  and  manages  product  technology 
needing  insight  as  it  matures  through  development.  The  critical  elements  are  required  to 
have  an  assessment  by  statutory  requirement.  Specifically,  this  is  referenced  in  DODI 
5000.02. 

Sullivan  (2010)  also  reported  the  welcoming  of  the  MRL  assessment  process  to 
improve  manufacturing  knowledge  and  to  improve  production  outcomes.  The  new  MRL 
process  was  to  consider  relevant  manufacturing  environment  development  in  a  similar 
manner  to  TRL  assessments  appropriate  to  the  life  cycle  acquisition  process.  Sullivan 
expressed  hope  that  after  14  years  of  reporting  negatively  about  production  outcomes  that 


57 


the  DOD  acquisition  process  would  realize  a  knowledge-based  approach  over  the  risk- 
based  approach  decried  by  the  GAO: 

A  serious  concern  is  that  DOD’s  in-house  manufacturing  workforce  has 
been  diminishing  for  decades  and  that,  therefore,  could  hamper  successful 
implementation  of  MRLs.  Unless  DOD  develops  long-range  plans  to  build 
its  in-house  manufacturing  workforce,  it  may  not  be  able  to  realize  the  full 
potential  of  integrating  manufacturing  readiness  levels  into  its  processes 
(47). 

and 

Direct  the  Office  of  the  Director,  Defense  Research  and  Engineering  to 
examine  strengthening  the  MRL  criteria  related  to  the  process  capability 
and  control  of  critical  components  and/or  interfaces  prior  to  MS  C,  or 
equivalent,  for  low-rate  initial  production  decision  (47). 

With  the  on-going  findings  of  the  GAO  and  the  review  of  the  DAS  documents,  a 
clear  picture  of  what  is  absent  in  DOD  manufacturing  knowledge  compared  to  non-DOD 
industries  reveals  weaknesses  in  the  DOD  approach.  Acquisition  practices  in  the  DOD  at 
MS  C  do  not  demonstrate  production  capability  or  producibility  performance  to  any 
specified  standards.  Therefore,  the  DOD  weapons  acquisition  process  fails  to  provide  a 
means  to  gain  confidence  in  a  contractor’s  ability  to  produce  a  given  design. 


58 


III.  PROJECT  APPROACH 


A.  VALUE-HIERARCHY  DEVELOPMENT  AND  THE  ICOM  MODEL 

Systems  engineering  methods  and  tools  can  help  define  a  development  path 
hoping  to  satisfy  a  system’s  requirements.  The  SE  approach  endeavors  to  establish  a 
value-hierarchy  representing  the  desired  state  of  process  or  product.  In  support  of  this 
thesis  SE  analysis,  there  are  simple  process  definition  tools  used  to  introduce  evaluation 
characteristics  for  the  more  detailed  AOA  discussed  in  Chapter  III,  Section  C. 

One  tool  that  helps  explore  this  space  is  the  input/controls/output/methods 
(ICOM)  model  (KBSI  2014).  The  ICOM  model  can  provide  a  visual  and  contextual  view 
of  the  DOD  production  approval  process.  The  content  of  the  current  DOD  process  is 
defined  by  the  manufacturing  development  and  approval  process.  This  can  be  defined 
further  by  the  development  of  the  DOD  acquisition  system’s  concept  of  operations 
(CONOPS)  giving  a  one  page  pictorial  view  of  a  system’s  broad  conceptual  operation. 
The  definition  of  simplified  ICOM  structure  of  the  DOD  manufacturing  development  and 
approval  process  is  represented  in  Figure  11. 


Regulations, 
Acquisition  Laws, 

Policy,  Contracts 


Figure  11.  Weapon  Systems  Acquisition  Process  Using  ICOM  Modeling 
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In  general,  the  ICOM  representation  provides  an  understanding  of  basic 
requirements,  methods  and  inputs  needed  to  describe  high  level  production  acquisition 
process  activities  and  show  output,  as  in  our  case,  as  either  a  future  effective  state  or  a 
current  ineffective  state.  At  this  stage  of  modeling,  the  ICOM  highlights  the  key  elements 
of  a  production  system  that  include:  manpower,  material,  methods,  and  machines. 

B.  REQUIREMENTS  DEFINITION  IN  THE  SEDP  METHOD 

System  engineering  practices  will  define  requirements  in  early  phases  of  a  project. 
These  requirements  are  refined  through  a  capture  of  stakeholder  and  user  needs  to 
understand  desired  customer  usage  and  avoid  potential  problems  in  system  definition. 
High-level  user  needs  and  lower-level  derived  requirements  are  developed  to  help 
achieve  a  robust  solution  against  competing  alternatives.  The  SEDP  process  follows 
guidance  found  in  a  published  handbook  given  by  the  Air  Force  Materiel  Command 
(AFMC  2014).  In  this  study  there  is  a  development  of  a  hierarchical  functional  valuing  of 
the  manufacturing  production  approval  processes  to  support  a  modeling  and  assessment 
objectively  contrast  alternative  best  practices. 

C.  AOA  USED  IN  THE  SEDP  APPROACH 

An  AOA  approach  provides  decision  makers  with  a  well-reasoned  assessment 
describing  the  factors  and  modeling  criteria  that  can  be  used  as  the  basis  for  selection  of  a 
preferred  alternative.  The  AOA  approach  guides  the  problem  solving  process  towards  a 
selection  from  alternatives  that  might  best  achieve  desired  capabilities.  The  preferred 
solution  can  emerge  out  of  a  complex  fog  of  potential  resources. 

The  comparative  approach  of  the  AOA  establishes  a  reasonable  story  showing 
how  best  practices  are  able  to  satisfy  a  preferred  process  outcome  from  the  alternatives 
under  study.  When  a  preferred  solution  emerges  from  an  AOA  there  is  an  opportunity  for 
decision  makers  to  consider  improvement  potential,  innovation,  cost  reduction  and  risk 
reduction  for  the  problems  they  are  trying  to  solve.  It  is  often  a  prudent  approach  in  the 
SE  selection  process  to  devise  a  verification  test  to  evaluate  any  proposed  solution 
confirming  the  preferred  alternative.  In  SEDP,  this  execution  step  is  an  initial  action  step. 
When  adopting  a  recommendation  from  and  AOA  some  tailoring  due  to  complexity,  cost 
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or  time  may  be  required  to  reach  a  better  alternative  that  balances  between  competing 
goals  of  cost,  performance  and  schedule. 

D.  STAKEHOLDERS 

This  study  recognizes  there  are  various  stakeholder  interests.  Identifying  and 
understanding  stakeholder  interest  leads  to  refinement  of  higher-level  requirements  and 
their  associated  value-functions  with  process  and  quality  objectives.  Identifying  these 
objectives  operationally  is  a  precursor  to  associating  appropriate  and  measurable  process 
best  practices.  The  development  of  measurable  objectives  to  achieve  improved 
production  outcomes  are  intended  to  satisfy  customer  expectations  as  developed  in  the 
CONOPS  discussion  in  this  chapter,  section  E. 

This  researcher  used  the  literature  review  and  experience  to  develop  a  mapping  of 
stakeholder  interests  to  manufacturing  related  business  practices.  This  mapping  is  used  to 
translate  stakeholder  values  into  process  objectives.  Process  objectives  can  be  related  to 
best  practices.  With  any  best  practice  there  is  a  presumption  of  quality  when  they  come 
from  successful  organizations.  In  other  words,  successful  organizations  achieve  their 
success  by  their  business  practices.  Using  the  best  practice  presumption,  one  can  assert 
that  if  an  industrial  practice  is  in  use  by  an  organization,  then  that  organization  derives 
value  with  its  practice  likely  satisfying  stakeholder  needs. 

Stakeholder  value  streams  can  be  associated  with  more  clarity  by  looking  into 
stakeholders  as  actors  and  with  transactional  roles  within  the  context  of  the  acquisition 
activities  in  production  part  approval  with  the  intention  of  achieving  ideal  production 
outcomes.  In  DOD  acquisition,  stakeholders  who  are  the  users  of  the  acquisition  products 
are  the  COCOM.  The  associated  operational  attributes  of  stakeholder  values  present  as 
suitability  factors  (Table  9). 
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Table  9.  Stakeholder  to  Process  Improvement  Suitability  Factors 


Stakeholder 

Suitability  Factors  for  Best  Practices 

Supplier 

Improved  business  efficiencies 

Supplier 

Improved  requirements  definition 

Supplier 

Improved  production  capability 

Supplier 

Improved  process  capability 

User/OEM 

Improved  production  outcomes 

OEM  &  Supplier 

Improved  customer  satisfaction 

In  defining  some  differences  between  DOD  and  non-DOD  stakeholders  there  are 
factors  specifically  related  to  the  production  approval  process  as  used  in  non-DOD 
practices.  Suitability  factors  for  production  approval  stakeholders  are  related  to  DOD 
acquisition  as  factors  important  to  end-item  users.  In  the  DOD  acquisition  environment 
prime  contractors  are  the  OEMs  and  in  non-DOD  acquisition  development  environments 
users  are  the  customers  who  purchase  products  from  OEMs. 

There  is  a  need  to  further  develop  the  identification  of  high  level  stakeholder 
values  to  measurable  objectives.  A  functional  hierarchy  can  describe  a  value  stream  from 
user  need  to  lower-level  functional  and  sub-functional  values  for  modeling  measurable 
attributes  as  measures  of  performance  (MOP).  Analytically,  the  lowest  measurable 
behavior  for  each  best  practice  becomes  a  key  measure  of  performance  for  scoring  in  the 
AOA.  The  creation  of  a  hierarchy  then  provides  the  means  to  assess  alternatives  and 
creates  elements  to  define  a  preferred  concept  of  operations  in  production  approval 
processes.  In  the  next  section,  a  discussion  of  stakeholders  and  their  relationship  to  the 
context  of  production  approval  use-cases  fully  form  an  operational  concept.  Two  cases 
will  be  described  to  explain  the  descriptive  and  normative  approval  process  CONOPs. 
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E.  OPERATIONAL  CONCEPT 


The  DOD’s  Architectural  Framework  (DODAF)  can  communicate  a  view  of  a 
system’s  CONOPS  at  a  glance.  The  system  level  view  of  key  transactional  relationships 
reveals  a  system’s  actors  in  their  interactive  behaviors.  A  complex  problem  under  the 
treatment  of  an  AOA  can  use  the  development  of  an  operational  concept  to  clarify  the 
descriptive  and  normative  use  cases  and  associated  process  outcomes.  The  variation 
found  in  alternatives  assists  the  robustness  of  an  AOA  assessment  of  study  alternatives  to 
satisfy  system  needs.  The  identification  of  process  attributes  and  MOPs  in  the  various 
approaches  provides  differentiating  criteria  found  in  an  alternative  and  its  ability  to 
provide  improvement  potential. 

With  the  identification  of  three  operationally  different  industrial  alternatives  the 
study  was  able  to  construct  for  comparison  the  operational  concepts  contrasting  non- 
DOD  to  DOD  production  approval  process.  These  alternatives  were  different  with  respect 
to  the  types  of  knowledge  required  to  make  production  decisions  for  their  associated 
commercial  production  processes.  The  DOD  approach  tended  to  focus  on  product 
technology  and  functional  requirements  and  the  non-DOD  organizations  had  an 
additional  focus  upon  manufacturing  development  and  demonstration  of  manufacturing 
capability.  Identifying  two  non-DOD  alternative  production  approval  processes  came 
from  regulatory  agencies:  1)  the  FAA  (commercial  aircraft  industry)  and  2)  the  FDA 
(medical  industry).  The  automobile  industry  was  a  third  commercial  industry  used  to 
compare  acquisition  alternatives  for  production  approval  processes. 

In  the  non-DOD  sectors,  the  part  suppliers  required  a  PSW  approval  allowing 
that  supplier  to  enter  production.  The  supplier  had  to  certify  that  an  item  demonstrated 
functional  and  manufacturing  process  capability  together  with  any  unique  requirements 
and  regulatory  compliance  prior  to  gaining  approval  to  market  those  products.  In  the 
automotive  industry  the  OEMs  directly  managed  the  requirements  for  compliance  in 
support  of  a  vehicle’s  production.  The  FDA  and  FAA,  as  regulatory  agencies,  govern 
compliance  by  law;  (“Appendix  C.  FAA  Memo”),  (“Appendix  I.  FDA  Procedures, 
Premarket  Approval”),  (“Appendix  N.  Understanding  the  Automotive  QMS”)  and 
(“Appendix  J.  FAA  Production  Approval  Process”). 
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The  non-DOD  industries  assure  production-approval  conformance  by  their 
certification  requirement.  The  requirements  did  not  allow  a  supplier  to  change  its 
manufacturing  process  or  location  of  manufacturing  without  reevaluation  of  the  changed 
process  and  recertification.  In  addition,  the  readiness  certification  required  a  compliant 
company’s  QMS  to  show  a  registration  to  their  QMS  by  a  third  party,  a  configuration 
managed  design  definition  for  product  and  process,  product  qualification,  a  working 
production  environment  and  a  process  capability  demonstration  as  determined  at 
production  rates.  In  contrast,  the  DOD  accepts  supply  network  parts  into  inventory  when 
that  supplier  passes  a  qualification  based  on  performance  of  a  part’s  functionally. 

The  DOD  production  approval  process  for  suppliers  asserts,  by  policy,  that  a  part 
can  enter  inventory  if  that  part  is  subject  to  a  functional  qualification  without  any 
verification  that  a  supplier  is  capable  to  produce  based  on  any  process  verification.  The 
significance  of  a  demonstrated  process  control  in  the  supply  network  is  to  achieve 
sustained  quality  compliance  and  prevent  poor  product  outcomes.  At  times,  the  DOD 
may  on  a  case-by-case  basis  include  additional  verifications  of  a  supplier’s  parts  through 
a  first  article  inspection  and  acceptance  test.  This  type  of  added  contractor  activity  relates 
to  unique  and  non-standard  purchase  agreements  with  their  suppliers. 

The  lack  of  a  standard  approach  to  production  approval  identifies  why  non-DOD 
producers  have  solved  the  issues  reported  by  the  GAO.  A  demonstrated  repeatable 
production  process  that  includes  the  entire  supply  network  ensures  the  desired  consistent 
production  outcomes.  In  general,  the  DOD  does  not  consider  the  supply  network  of  the 
prime  contractor  beyond  part  qualification  unless  by  exception. 

1.  The  Current  DOD  Production  Approval  Process  CONOP 

Developing  the  graphical  representation  of  the  DOD  production  approval  process 
defines  a  depiction  of  the  actors  and  activities  with  communications  and  decisions  points. 
This  operational  view  (OV)  provides  a  means  to  describe  the  interworking  and  execution 
of  the  DOD  production  approval  process.  The  high-level  contextual  operational  view, 
OV-1,  provides  a  visualization  of  a  system  in  a  static  sense  with  various  nodal 
interactions,  activities  and  actors.  This  study’s  DODAF  OV-1  follows  a  sequence  starting 
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at  the  top  and  moving  counter-clockwise  to  show  the  life-cycle  acquisition  process  of  the 
activities  leading  up  to  production  approval  and  concluding  with  FRP. 

The  current  state  of  the  DOD  production  approval  process  is  referred  to  as  the 
descriptive  state  of  operations  of  the  production  approval  process  in  review.  This 
acquisition  environment  shown  by  the  OV-1  captures  the  actors  with  key  organizational 
elements.  The  transactional  nature  of  program  management  includes  the  production 
approval  event  at  MS  C  with  the  milestone  decision  authority  involvement. 

The  DOD  acquisition  technical  review  event  that  considers  production  readiness 
is  the  PRR  technical  review  conducted  just  prior  to  MS  C.  The  PRR  assesses  the  pre- 
production  baseline.  This  approach  has  a  high  degree  of  variation  in  the  absence  of 
formal  standards  and  relies  on  a  checklist  approach  that  has  inherent  weaknesses  if  not 
outright  omission  in  determining  a  given  process  capability.  The  results  of  a  PRR  reports 
manufacturing  risk  to  the  PM  and  the  PM  may  inform  the  CPD  about  manufacturing  risk. 
The  CPD  is  the  reviewable  document  given  to  the  Defense  Acquisition  Board  (DAB)  to 
affect  production  approval. 

The  PRR  review  is  to  assure  that  a  production  readiness  has  been  achieved  with 
any  reported  risk  when  a  program  is  to  transition  from  the  EMD  to  the  PD  phase.  The 
information  used  to  formulate  the  graphical  OV-1  came  from  Table  10.  See  Figure  12. 
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Table  10.  Descriptive — DOD  Actors  /  Stakeholders  DOD  Roles  and 

Activities 


Actor 

Organization 

Transaction 

Congress 

Congress 

Funding;  Law;  Oversight 

Sponsor;  User; 
Fleet;  Government 
Acquisition 

OT&E/OPTVFORE/ 

COCOM 

Manage  program/funding;  Apply  JCIDS  -  KPPs  / 
KSAs->  COIs/MOEs/TPMs);  Use  -Case  for  System  for 
fulfilling  unrealized  capability  need  Evaluate 

Suitability  and  Effectiveness  to  meet  the  need  -  Present 
to  Congress 

OEM  - 

Producer/Prime 

Contractor 

From  MS  B  -  Provide  Development  of  Product  and 
Process  (TMRR/EMD);  Establish  product  baseline  and 
provide  readiness  to  produce  for  LRIP/FRP  at  MS  C  - 
Risk-based  Acquisition. 

Maintainer  I/O/D 

Flight  Line/Logistics 

Contractor 

Provide  Support  Services  for  Product  In  Operational 
Environment/Intermediate;  Organizational;  Depot 
Services 

Congress  /OSD  / 
Milestone 

Decision  Authority 

Acquisition  Authority 

Provide  Funding;  JCIDS  High-level  requirements  - 
KPP;  KSA;  Periodic  Program  Approval 

Supply  Network 

Supplier  tiers 

Provides  items/services  to  prime;  qualified  parts 
required. 

rd 

3  Party  Auditors 
DCCA;  DCMA 
Authority 

QMS  Auditor; 

Contracts 

Assessment  of  Compliance;  e.g.  AS9100 

Completes  DD250  to  approve  of  payment  for 
compliant  deliveries 

Panel  Members 

Executives 

PM  and 
Program  Office 

Technical 

Manager 

Authorities 

Contractor 

Executives 

COCOM 

Government 

Executives 

Provide  Recommendation  to  decision  authorities  at 
Project  Milestones;  Provide  Project  Oversight  at 
Technical  Reviews 

Here,  a  program  determines  if  requirements  from  the  JCIDS  process  have  been 
satisfied.  The  CPD  is  the  document  that  reports  the  support  for  a  decision  to  advance  to 
production.  In  brief,  these  requirements  cover:  1)  production  items  that  are  purchased  at 
the  contracted  price  and  delivery  schedule  and  2)  production  items  that  meet  the  design 
requirements  through  qualification.  The  end  goal  of  the  DOD  production  approval 
process  is  to  ensure  that  items  produced  are  suitable  and  effective  as  observed  by  the 
Operation  and  Test  Evaluation  (OT&E)  and  qualification. 
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If  approval  to  start  production  is  to  be  granted  then  it  occurs  at  MS  C.  This  does 
not  include  a  production  capability  demonstration.  The  OV-1  shows  that  it  is  not  until 
after  production  has  started  at  the  end  of  LRIP  that  a  PCA  would  support  a  demonstrated 
production  capability  assessment. 

The  DOD  as-is  state  OV-1  shows  that  development  of  the  production  system 
continues  into  PD  through  LRIP.  Therefore,  it  is  uncertain  that  the  LRIP  experience  will 
actually  mature  the  manufacturing  environment  and  be  able  to  demonstrate  a  production 
rate  and  capability  to  support  an  FRP.  Any  deficiency  found  at  the  PCA  will  hinder 
production  into  FRP  with  its  associated  impact  on  fleet  readiness  and  effectiveness.  This 
uncertainty  is  quite  different  when  applying  best  practices  as  observed  in  non-DOD 
production  approval  processes  that  certify  the  ability  to  produce  prior  to  production  start. 


Technology 

Readiness 


Manufacturing 

Readiness 


TMRR  Phase 
(PDR) 


P.  L.  TRL  6  Required 
MRL  6  Estimated 
Relevant _ 


f  MSA  Phase  j 


PD  Phase  -  frp 


M  DA 


PD  Phase  -  LRIP 

(PCA) 


Production  Demonstration 


Program 

Management 


EMD  Phase 

(Critical  Design  Review)  ->(Production  Readiness  Review) 


Component 


Syste 


Qualification 


Qualification 


MRL  7  -  Representative  Line  ->  MRL  8:  Pilot  Line 


Figure  12.  (OV-1)  DOD  Descriptive  (Current)  State 


2.  The  Non-DOD  Production  Approval  Processes 

The  assessment  of  actors  and  roles  in  the  automotive  acquisition  process  for 
production  approval  uses  the  prescriptive  ISO  /  TS- 16949  QMS  as  applied  across  the 
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supply  base.  The  FAA  and  FDA  use  an  agency-acceptable  QMS  such  as  the  AS-9100  or 
ISO-9000  standards.  The  FDA  and  FAA  append  the  addition  of  regulatory  specific 
requirements  and  a  demonstration  to  production  approval  requirements  making  their 
production  approval  processes  more  like  the  automotive  process.  The  intended 
production  capability  is  resolved  by  the  use  of  the  non-DOD  prescriptive  standards  and 
requirements  following  their  QMS.  Contextual  information  is  shown  from  the  automotive 
perspective  to  create  the  transactional  relationships  from  their  best  practices.  See  Table 
11.  These  transactions  describe  how  compliance  for  production  readiness  leads  to  a 
production  decision  in  non-DOD  industries. 

The  description  of  actors,  organizations  and  transactions  for  the  automotive 
industry  production  approval  process  is  very  similar  to  the  other  non-DOD  production 
approval  processes.  The  context,  shown  by  the  automotive  production  approval  process 
develops  and  demonstrates  production  readiness  from  their  best  practices  from  a 
disciplined  manufacturing  development  and  demonstration  approach  using  their  industry 
standards.  These  unique  non-DOD  acquisition  elements  capture  the  knowledge-based 
production  approval  processes  observed  by  the  GAO  reporting  (Sullivan  2008). 


Table  11.  Automotive  Actors  and  Activities  Using  a  PPAP  Certification 


Actor 

Organization 

Transaction 

Producer 

Original  equipment 
manufacturers 

Requirements;  Integrator;  Producer;  Sell/ 
Warrant  /Service;  Product  Sold  to  Users 

User/ 

Customer 

Individual  or  Organization 

Buy  /  Use  Vehicle 

Dealer  - 
Service 

OEM  or  Independent 

Sell  /  Service  Vehicle 

Government  / 
Legal 

National  Transportation  and 
Safety  Board 

Requirements 

Homologation; 

Regulations 

Supplier 

Producer 

Component  /  Subsystem 
Supplier  Tier  1  -  to  OEM 

Design  Responsible  ;  APQP;  PPAP; 
Delivery 

Supply 

Network 

Tier  2;  Tier  3  and  ... 

Supplies  to  Tier  1 
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Actor 

Organization 

Transaction 

rd 

3  Party 
Certifier 

Certification  Bodies  -  Quality 
System; 

OEM  Directed  Compliance  to  Supply 
Network 

rd 

3  Party 
Training 

AIAG/SAE/ASQ/SE 

Related  Training;  supply  base  enablers 

3.  Normative  DOD  Production  Approval  Process 

A  normative  state  production  approval  process  for  DOD  implementation 
considers  the  integration  of  non-DOD  best  practices.  The  transactional  actors  and  actions 
are  described  by  synthesizing  necessary  elements  of  the  current  DOD  process  and 
merging  non-DOD  production  approval  processes.  This  normative  (or  ideal)  state  for 
DOD  production  approval  would  then  describe  actors,  organizations  and  transactions  of 
both  QMS  based  systems.  The  modifications  formed  in  this  way  use  the  current  DOD 
actors  of  Table  10  and  those  best  practices  of  non-DOD  industries  found  in  Table  11.  The 
combined  actors  and  transactional  behavior  from  the  synthesis  and  integration  of 
commercial  best  practices  with  the  DOD  transactions  are  in  Table  12. 


Table  12.  Normative — DOD  Actors  /  Stakeholders  DOD  Roles  and 

Activities 


Actor 

Organization 

Transaction 

Comment 

Congress  /OSD  / 
MDA 

Acquisition 

Authorities 

Provide  Funding;  JCIDS  Fligh-level 
requirements  determined  -  KPP;  KSA; 
Periodic  Program  Approvals 

Same 

Sponsor;  User; 
Fleet  (Customer) 

OT&E/OPTVFO 

RE/COCOM 

Government 

Acquisition 

Strategy  & 

Oversight 

Manage  program/funding;  Apply  JCIDS  - 
KPPs  /  KSAs->  COIs/MOEs/TPMs);  Use 
-Case  for  System  for  fulfilling  unrealized 
capability  need  Evaluate  Suitability  and 
Effectiveness  to  meet  the  need. 
Knowledge-Based  Acquisition. 

Modified 
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Actor 

Organization 

Transaction 

Comment 

Integrating 
Contractor  - 
Producer/Prime 

Contractor 

Provide  Development  of  Product  and 
Process  -  Technical  Maturity  and  Risk 
Reduction  and  (EMD);  Establish  product 
baselines  and  provide  demonstrated 
process  capability  at  MS  C,  follow  PPAP, 
LRIP,FRP 

Modified 

Supply  Network 

Tier  1;  Tier  2 
and  . . . 

Supply  network  provides  evidence  for 
APQP  and  PPAP  -  to  higher  tier.  Tier  1  to 
Prime  Contractor  PPAP. 

Modified 

3rd  Party  Certifier 

Certification 
Bodies  -  Quality 
System 

OEM  Directed  Compliance  to  Supply 
Network 

Modified 

3rd  Party 

Training 

AIAG/SAE/ASO/ 

SE 

Related  Training;  supply  base  enablers 

New 

Legal  /  Agency 

Certifications; 

Boards 

Requirements  Demonstration,  Report  to 
decision  authorities 

Modified 

1st  Party 

Auditors 

DCCA;  DCMA 

Agency, 

Contract 

Administration 

Assessment  of  Contract  Compliance; 
Completes  DD250  to  approve  of  payment 
for  compliant  deliveries;  Oversight 

Same 

Panel  Members; 
Program 

Manager  and 
Program  Office; 
Technical 

Manager 

Government: 
COCOM; 
(PMA)/(PEO)  & 
Subject  Matter 
Experts 

Contractor 

Provide  Recommendation  to  decision 
authorities  at  Project  Milestones;  Provide 
Project  Oversight  at  Technical  Reviews 

Same 

Maintainer  I/O/D 
(Government) 

Flight 

Line/Logistics/ 

COCOM 

Provide  Support  Services  for  Product  In 
Operational  Environment/Intermediate; 
Organizational;  Depot  Services 

Same 

Maintainer  I/O/D 
Supply  Network 

Contractor 

Provide  Support  Services  for  Product  In 
Operational  Environment/Intermediate; 
Organizational;  Depot  Services 

Provides  items/services  to  prime;  Follows 
PPAP  -  as  needed  in  PD.  Follow 

ISO/TS 16949  /AS 9 145  PPAP/APQP 

Modified 

The  integration  of  non-DOD  best  practices  into  the  DOD  acquisition  process 
visually  represents  the  normative  operational  concept.  The  normative  operational  view,  at 
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a  glance  is  shown  in  Figure  13.  It  is  important  to  point  out  that  the  knowledge-based 
manufacturing  method  recommended  by  the  GAO  is  fully  captured  in  the  synthesis  of 
DOD  with  non-DOD  standards  providing  a  production  commitment  at  MS  C. 

The  normative  model  reveals  key  changes  over  the  current  state  DOD  OV-1  by 
pulling  the  DOD  PCA  ahead  prior  to  MS  C.  The  PRR,  in  EMD,  would  then  include  the 
demonstration  of  process  capability  using  the  DOD  PCA  demonstration  of  process 
capability  prior  to  MS  C.  This  would  pull  the  nature  of  the  MRL  ratings  to  a  higher  level 
of  MRL  9  at  the  end  of  EMD.  The  necessary  evidence  of  a  functional  production  line  at 
the  end  of  EMD  incorporates  a  full  development  and  verification  of  the  production  line 
prior  to  production  commitment  at  the  end  of  EMD. 

This  CONOPS  review  sets  up  the  development  and  analysis  conducted  in  the 
thesis  AOA.  This  synthesis  would  likely  have  to  assure  that  development  funding 
incorporates  the  manufacturing  development  fully  prior  to  LRIP  to  launch  a  completed 
production  line.  In  the  alternative  the  full  manufacturing  development  and  demonstration 
could  require  long  lead  funding  of  a  production  contract  that  is  pulled  into  EMD  to  build 
production  items  in  the  pre-PD  phase  rather  than  using  LRIP  to  develop  the  final  FRP 
manufacturing  process. 
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MRL7/8  -  Representative/Pilot  Line  ->  MRL  9:  Production  Line 


Figure  13.  (OV-1)  DOD  Normative  (Ideal)  State 


F.  BEST  PRACTICE  BEHAVIORS 

The  AO  A  assessment  of  alternative  production  part  approval  processes  fulfils  a 
need  in  DOD  acquisition  to  consider  improving  the  current  culture  that  has  prevented  the 
implementation  of  best  practice  opportunities.  Specifically,  if  an  organization  wants  to 
improve  it  must  have  a  bias  to  improve  business  operations  and  do  so  purposefully. 
Gardner  (2004)  “The  Process-Focused  Organization,”  indicates  that  the  creation  of  a 
policy  group  or  best  practice  council  is  an  effective  means  to  identify  and  manage 
continuous  improvement.  Gardner  further  explains  that  an  organization  needs  to  look 
across  and  outside  organizational  lines  to  conduct  best  practices  research. 

This  is  the  same  notion  the  GAO  asserted  indicating  that  the  DOD  should  look  to 
industries  outside  the  DOD  that  were  more  successful  because  they  had  demonstrated  a 
followed  a  more  disciplined  approach  to  production  development  and  demonstration  of 
production  capability  prior  to  production  commitment  (Schwartz  2013).  Additionally, 
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Gardner  (chapter  9.7),  describes  a  two-part  strategy — first  to  research  potential  best 
practices  and  secondly  to  provide  an  evaluation  and  implementation  activity.  Gardner’s 
approach  is  in  agreement  with  the  SEDP  approach  to  problem  solving.  Improvement 
opportunity  needs  to  be  ingrained  in  organizational  culture  by  including  and  empowering 
an  improvement  council  to  provide  a  way  to  evaluate  and  manage  change.  Each  of  the 
non-DOD  best  practices  was  at  some  point  identified  and  was  found  to  contribute  to 
operational  success  and  business  improvements.  This  aligns  with  the  SAE’s  purpose  to 
serve  its  members  by  developing  standards  to  improve  the  industries  they  serve. 
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IV.  ANALYSIS  OF  ALTERNATIVES 


A.  DEVELOPING  THE  STUDY  AOA  MODEL 

The  study  AOA  forms  a  part  of  a  SE  problem  solving  effort  examining  a  process 
that  needs  improvement  in  DOD  acquisition  practices  with  respect  to  production 
approval.  This  AOA  captures  an  improvement  potential  by  comparing  the  alternate 
production  approval  processes  of  the  FDA,  FAA,  and  automotive  industry  to  the  DOD 
approach  and  assesses  industry  best  practices  and  that  have  shown  superior  results.  Using 
the  Air  Force  AOA  handbook  helped  to  model  the  assessment  of  best  practices  as  found 
in  an  industry’s  QMS  (such  as  the  QS-9000  family  of  standards  for  the  automotive 
industry),  those  practices  of  the  FAA  (aircraft  sector),  and  the  FDA  (medical  devices 
sector)  (AFMC  2014).  QMS  best  practices  were  identified  and  described  by  each  of  the 
study  QMS  types  as  ISO-9000  non-prescriptive  and  QS-9000  prescriptive  QMS.  The 
FAA  and  FDA  regulatory  process  construct  has  been  described  by  their  respective  federal 
regulations  are  found  on  agency  websites.  From  the  FAA  website  -  FAA.gov  and  the 
FDA  website  FDA.gov  respectively  a  high  degree  of  fidelity  their  production  approval 
processes  are  outlined.  The  government  websites  contained  the  relevant  information  in 
support  of  a  comparative  analysis  needed  for  the  AOA. 

B.  QUALITY  MANAGEMENT  STANDARDS  COMPARISON 

The  differences  between  QMS  approaches  of  ISO-9000  and  QS-9000  were 
significant  for  the  unique  U.S.  requirements  driven  by  Ford,  General  Motors  and  Chrysler 
and  the  more  global  ISO  standards  development.  The  QS-9000  included  additional 
business  processes  such  as  planning,  customer  satisfaction,  continuous  improvement, 
manufacturing  capabilities  and  advanced  quality  planning.  The  automotive  industry 
needed  a  uniformly  prescriptive  set  of  instructions  as  best  practices  using  their  experience 
and  influence  to  advantage.  The  desired  improvements  in  manufacturing  capabilities 
needed  to  be  confirmed  by  a  PPAP  submission  of  a  warrant  prior  to  production.  The 
American  OEMs  also  knew  that  improved  production  outcomes  needed  manufacturing 
development  best  practices  as  an  enabling  companion  to  the  PPAP.  The  development 
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standards  for  manufacturing  employed  the  APQP,  a  synergistic  partner  communicated  by 
guidelines  as  advanced  by  the  AIAG  consortium. 

The  aerospace  and  defense  sectors  adopted  the  ISO-9001  standard  and  embedded 
it  into  the  aerospace  QMS  AS-9000  and  conforming  later  the  release  of  the  AS-9100 
QMS.  Aerospace  (including  National  Aeronautics  and  Space  Administration,  Aerospace, 
and  Defense)  unified  their  approach,  but  unlike  the  automotive  QMS,  the  aerospace 
standards  stayed  more  aligned  to  the  generic  best  practices  found  in  the  ISO-9000. 

The  DOD  could  invoke  by  contract  the  use  of  the  AS-9100  QMS  for  prime 
contractors  but  did  little  to  standardize  this  application  of  a  QMS  in  supply  networks 
leaving  supply  chain  management  to  each  prime  contractor.  The  DOD  did  not  require  any 
customer  specific  or  prescriptive  practices  differentiating  DOD  contractors  as  adherents 
of  the  ISO-9000  type  of  QMS. 

The  adoption  of  the  QS-9000  standards  as  a  requirement  for  suppliers  found  over 
time  that  there  were  considerable  quality,  economic  and  customer  satisfaction  gains.  The 
AIAG  and  the  ASQ  had  an  interest  in  measuring  the  impact  of  the  constituent  supplier 
application  in  the  new  automotive  QMS.  Specifically  they  wanted  to  measure  if  the 
OEM’s  quest  to  improve  product  quality  and  regain  market  share  through  applying  the 
QS-9000  QMS  was  working.  Many  participating  organizations  were  interviewed  and 
captured  these  results  in  an  ASQ  survey.  This  survey  was  conducted  ten  years  after  QS- 
9000  introduction  in  1997  with  a  confirming  follow-on  survey  in  1998. 

Subsequently  the  McGraw-Hill  company  conducted  a  survey  of  ISO-9000 
companies  (comparable  to  the  DOD  AS9100  QMS  application).  The  comparison  and 
basic  affect  is  given  in  stark  contrast  showing  significant  differences  in  quality  in 
application  of  those  suppliers  adopting  the  QS-9000  standards  and  the  McGraw-Hill 
survey  (see  Table  13)  (AIAG/ASQ  1997;Naveh,  Marcus  and  Koo  Moon  1999). 

The  automotive  industry  standards  established  a  common  product  and  process 
development  language  between  OEM  and  supplier.  There  were  common  expectations 
following  the  common  QMS  for  design  and  production  authority  within  the  supply  chain 
at  the  global  sourcing  level.  OEMs  were  able  to  have  requirements  flow  down  to  supplier 
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organizations  dendritically.  The  intent  to  have  one  QMS  identified  with  proven  methods 
paid-off.  Suppliers  were  required  to  satisfy  OEM  minimum  requirements  and  submit 
evidence  that  products  would  meet  product  and  production  performance  targets  prior  to 
an  OEM’s  production  commitment. 


Table  13.  Survey  Benefit  Results — ISO  Vice  QS-9000  QMS 


Survey 

Number  of 
Respondent 

Cost  to 
Register 

Maintenance 

Costs 

Economic  Benefits 

Quality  Defect  Rate 

QS  9000 

Combined 

$120,000 

$40,000 

3  to  1  return  on  total 

costs 

50%  of  respondents  50% 
reduction  in  defect  rate  due  to 

ASQ/AIAG1997  & 
1998 

800 

improvement  6%  of 
sales 

QS  9000 

ISO  9000 

1100 

$156,000 

Unreported 

1 .3  to  1  return  on  total 

6%  of  respondents 
indicated  a  reduction  in  defect  rate 

due  to  ISO  9000 

McGraw-Hill  1999 

cost 

Member  companies  were  required  to  conduct  a  third-party  audit  to  establish 
compliance  costing  an  average  of  $40,000;  see  Table  13.  Bandyopadhyay  (1996,  12) 
confirmed  this  approximate  fee  averaged  about  $50,000  to  register  including  most 
preparation  fees  not  expressly  recorded  in  the  survey  findings. 

1.  Statistical  Significance  and  Quality  Standard  Type 

If  an  organization  is  to  seek  to  improve  business  processes  then  various 
alternatives  can  be  competed  to  find  a  better  approach.  A  selection  between  two 
processes  could  be  evaluated  by  a  comparison  of  two  treatments.  If  each  treatment  had 
many  trials  that  expressed  results  as  success  or  failure  then  a  best  process  could  emerge 
based  on  this  performance.  In  this  study  of  production  approval  process  alternatives  the 
ISO-9000  QMS  as  one  treatment  -  characterized  by  its  non-prescriptive  approach  and  the 
QS-9000  QMS  as  a  second  treatment  -  characterized  by  its  prescriptive  approach.  The 
DOD  acquisition  approach  considers  a  prime  contractor  that  adheres  to  the  ISO-9000  or 
AS-9100  type  QMS.  Then  non-DOD  organizations  that  use  a  prescriptive  QMS  like  the 
automotive  QS-9000  or  the  regulated  FAA  and  FDA  acquisition  processes. 
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To  contrast  the  survey  results  the  key  question  asked  in  each  trial  group  (survey 
respondent)  was  to  assess  if  there  was  an  improvement  observed  under  the  treatment 
application  of  their  respective  QMS  types.  The  ISO  QMS  type  showed  quality 
improvement  in  just  6%  of  1100  cases.  In  the  Automotive  type  QMS  they  recorded  an 
improvement  in  50%  of  respondents  in  800  cases.  The  results  of  these  trials  were  able  to 
show  a  significant  difference  with  the  automotive  QS-9000  QMS  had  better  quality  results 
than  the  ISO-9000  QMS.  In  this  way,  a  reviewer  could  discriminate  between  processes  if 
the  differences  observed  were  sufficient  to  recommend  one  treatment  as  better  than  the 
other  where  the  differences  were  not  due  to  randomness  and  change  from  sampling  alone. 
Data  was  analyzed  from  the  commercial  surveys  providing  the  production  outcomes  by 
QMS  type.  A  test  of  significance  at  the  (1  -  .05  alpha)  level  was  conducted.  See  Table  14. 
The  results  confirm  that  there  is  a  significant  difference  observed  at  the  alpha  .05  level  with 
a  p-value  of  0.000.  Those  respondents  that  used  the  ISO-9000  standard  of  Treatment  A 
versus  those  that  used  the  QS-9000  of  treatment  B  were  found  different  favoring  the  QS- 
9000  QMS  as  providing  superior  quality  improvement. 


Table  14.  Survey  Benefit  Results — ISO  Vice  QS-9000  Test  of  Significance 


Two  Sample  Proportions  -  TEST 

ISO-9000  QMS  vs  QS-9000  QMS 

P  value  at  the  alpha  .05  level  =  0.000 

Quality 

System 

Sample  Size 

Companies 

Reporting 

Improvement 

Fraction  of 
Companies 
Improved  - 
Sample  (P) 

Survey  Source 

Treatment  A 
ISO-9000* 

1100 

66 

0.06 

McGraw-Hill  1999 

Treatment  B 
QS-9000** 

800 

400 

0.5 

AIAG/ASQ  1997/8 

Note:  95%  Cl  for  p(QS)  -  p(ISO):  (0.402618,  0.477382)  Estimate  for  p(QS)  -  p(ISO):  0.44 

Test  for  p(QS)  -  p(ISO)  =  0  (vs  not  =  0):  Z  =  23.07  P- Value  =  0.000 

*  ISO  companies  reported  only  6%  improved  due  to  ISO  9000  as  their  QMS  (no  degree  of  improvement  reported) 

**  QS  companies  reported  that  50%  improved  at  least  50%  due  to  QS  9000  as  their  QMS 
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In  any  survey  study  it  is  important  to  make  an  assessment  regarding  validity  and 
reliability.  Environmental  factors  show  that  each  QMS  matured  in  use  about  ten  years. 
Each  QMS  type  standard  had  at  its  core  the  same  ISO  9000  elements.  The  primary 
difference  between  QMS  standards  was  the  enhancement  defined  by  the  guidance  of  the 
APQP  and  requirement  of  a  PPAP.  The  PPAP  used  SPC  to  demonstrated  process 
capability  prior  to  production  commitment.  In  addition,  there  was  a  global  nature  of  each 
supply  network  surveyed  with  the  ISO  9000  type  QMS  favoring  more  European 
respondents  with  both  QMS  used  in  a  global  economy.  The  repeat  nature  of  the  QS-9000 
survey  showed  similar,  in  position,  results  of  respondents  being  just  separated  by  time. 
Therefore,  the  commercial  surveys  cited  were  found  to  have  validity  and  reliability. 

2.  Benefits  of  Prescriptive  Quality  Standards 

The  published  surveys  by  ASQ/AIAG  and  McGraw  Hill  indicated  that  the 
automotive  industry’s  ISO  enhanced  standards  (QS-9000  /ISO/TS- 16949)  performed 
remarkably  better  with  respect  to  quality  outcomes  than  the  ISO  based  QMS  as  used  in 
the  DOD.  The  QS-9000  based  QMS  surveys  revealed  a  benefit  analysis  captured  in  a 
presentation  by  Ireland  (2000),  citing  that  AIAG/ASQ  1998  Quality  Survey,  “Cost 
Benefit  Summary  with  600  Respondents”: 

Of  the  600  survey  respondents  using  the  QS-9000  automotive  quality 
management  systems  standards  it  was  reported  that  sales  increased  by  6% 
or  $10,000,000  per  company  on  average.  The  supply  network  applied  the 
quality  standards,  received  third  party  registration  and  followed  the 
production  approval  approach  of  the  automotive  OEMs  (Ireland  2000). 

Ireland  also  reported  (2009)  that  there  was  an  increase  in  vehicle  durability  over 
that  same  period  where  vehicles  improved  their  durability  by  more  than  1.5  years;  now 
averaging  8  years  (Ireland  2009).  USA  Today  agreed  with  Ireland  in  that  the  automotive 
vehicle  reliability  was  increasing  and  that  it  continued  to  increase  showing  the  average 
age  of  a  vehicle  on  the  road  now  exceeds  1 1.4  years  (Bomey  2015). 

Further  study  on  vehicle  durability  over  time  showed  that  there  was  a  correlation 
aligned  to  the  adoption  of  the  QS-9000  family  of  standards  and  the  data  published  from 
the  Office  of  the  Assistant  Secretary  for  Research  and  Technology,  Bureau  of 
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Transportation  Statistics  ,  (USDTOASR  &  TBOT),  showing  calendar  year  versus  average 
age  of  vehicle  on  the  road  from  1980  to  2000  (USDTOASR  &  TBOT  2014).  What  is 
noticeable  is  that  durability  growth  was  flat  from  1985  to  1990  prior  to  QS-9000.  There 
was  a  continuation  of  sustained  growth  through  2000  based  on  the  DOT  reporting.  The 
linear  best  line  fit  to  the  DOT  age  data  from  1990  to  2000  can  be  projected  at  this  same 
rate  of  growth  linearly  through  2015  (Figure  14). 

The  nature  of  these  results  shows  mathematical  correlation  that  the  production 
approval  process  for  the  automotive  industry  was  supported.  Ireland  (2009)  reported  that 
there  were  a  number  of  organizational  benefits  from  the  use  of  the  automotive  standards: 

•  Suppliers  realized  cost  efficiencies  by  the  use  of  a  common  standard 

•  Suppliers  realized  product  quality  and  reliability  improvement 

•  OEM  realized  vehicle  quality  and  reliability  improvement 

•  OEM  realized  cost  efficiencies  by  use  of  a  common  standard,  23 

There  were  no  similar  quality  improvements  reported  by  the  McGraw-Hill  survey 
under  the  use  of  the  ISO-9000  standard  from  1100  respondents  (Naveh,  Marcus  and 
Moon  1999).  The  benefits  of  a  prescriptive  QMS  as  used  in  the  auto  industry  provided 
the  compelling  reason  for  the  development  of  a  detailed  functional  analysis  for  AOA 
comparing  alternate  production  approval  processes. 
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Figure  14.  Automobile  Durability  Average  No.  of  Years  on  the  Road.  Adapted 

from  USDTOASR&TBOT  (2014). 


C.  FUNCTIONAL  ANALYSIS 

Information  concerning  stakeholder  needs  leads  to  an  identification  of  functional 
values  and  sub-values  that  develop  discrimination  criteria  for  the  exercise  of  the  AOA. 
The  functional  analysis  helps  define  factors  of  a  measurable  effectiveness  against 
identifiable  alternative  best  practice  behaviors. 

A  process  of  functional  analysis  starts  with  the  higher-level  requirements  in 
stakeholder  needs.  Building  upon  the  simple  ICOM  model  the  additional  description  with 
the  DOD  acquisition  process  can  show  the  current  and  ideal  state  output  (Figure  15).  The 
output  side  of  the  ICOM  model  shows  that  there  is  a  difference  in  expectations  for  poor 
and  ideal  production  performance.  By  using  the  findings  from  the  AIAG/ASQ  and 
McGraw-Hill  QMS  surveys  a  case  supporting  a  normative  outcome  was  developed. 

The  underlying  performance  improvement  in  the  automotive  industry  best 
practices  is  traced  to  the  use  of  prescriptive  quality  standards.  Production  approval 
process  alternatives  could  then  address  the  gap  between  descriptive  and  the  normative 
production  process  outcomes.  Consider  the  supporting  quote:  “It  must  be  possible  to 
estimate  how  well  a  system  is  doing  in  its  drive  toward  the  goal  or  how  closely  one 
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option  or  another  approaches  the  ideal”  (Gibson,  Scherer,  and  Gibson,  2007,  2).  As 
shown  in  the  ICOM  diagram  and  the  OV-1  model  there  is  a  contrast  of  results  by 
organizational  QMS  type.  See  the  descriptive  OV-1  in  Figure  12  and  the  normative  OV-1 
in  Figure  13. 


Acquisition 

Policy, 


Regulations, 

Laws, 

Contracts 


Industry 


Figure  15.  ICOM  Model  with  Normative  and  Descriptive  Output. 


The  value  system  for  a  production  approval  process  has  its  genesis  in  the  GAO’s 
reporting  of  the  manufacturing  related  causal  factors  for  poor  production  outcomes. 
When  examining  the  manufacturing  causal  factors  the  study  identified  six  key  objectives 
for  evaluation.  See  Table  15.  The  study  value  system  of  hierarchical  functions  and  sub¬ 
functions  are  reviewable  starting  in  Figure  16.  The  value  system  continues  as  derivative 
elements  into  measurable  evaluation  factors  within  the  value  stream  establishing 
increasing  fidelity  into  sub-functions.  See  Figure  17.  For  completeness  of  the  study 
problem  space,  the  shaded  “gray  box”  items  include  notionally  the  two  additional  causal 
factors  reported  by  the  GAO  related  to  poor  production  outcomes  as  excluded  items  in 
the  study. 

A  structure  tracing  six  binning  categories  for  the  higher-level  objectives  to 
various  counting  lower- level  measurable  sub-functions  under  each  bin  that  applies.  These 
lower  level  objectives  are  the  countable  measures  of  performance  aligned  to  individual 
best  practices  in  industry  production  approval  processes.  The  element  as  a  measurable 
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best  practice  under  value  element  1.1  is  “QMS  ISO-9000;  AS-9100;  ISO/TS-16949  w/ 
Certification”  (Figure  17).  The  element  1.1  is  then  the  lowest  measure  of  performance 
(MOP)  under  study  as  a  measurable  objective.  Reviewing  all  measurable  objectives  starts 
with  objective  1.0  through  objective  6.0  showing  37  identifiable  evaluation  items  in  the 
AOA.  The  MOP  elements  define  a  discrete  listing  of  best  practices  in  production  part 
approval  processes.  The  study  MOPs  are  the  countable  number  of  individual  best 
practices  aligned  in  relation  to  the  six  objectives.  For  all  37  MOPs;  see  Figure  17  through 
Figure  22.  Definitions  for  each  measurable  best  practice  MOP  is  found  in  the  appendix. 
For  more  information  see  “Appendix  K.  MOP  Rationale  and  Weighting  Factors.” 

The  AOA’s  ability  to  discriminate  between  alternatives  under  review  is  by  the  use 
of  the  six  objectives  defined  as  logical  groupings  of  the  countable  best  practices  found  in 
the  non-DOD  industry  practices.  These  practices  are  articulated  in  the  QS-9000  (ISO/TS- 
16949)  family  of  automotive  standards  and  guidance  manuals.  The  reason  these  MOPs 
represent  the  normative  best  practices  is  from  the  accounting  that  these  quality  system 
standards  produced  the  best  production  outcomes  observed  by  the  GAO  and  the 
assessments  from  the  commercial  surveys  in  review.  These  best  practices  align  to  the  1 1 
normative  production  part  approval  best  practice  attributes  and  represent  the  prescriptive 
quality  system  captured  in  stakeholder  needs  (Table  15). 


Table  15.  SEDP  Objective  Binning  for  Countable  Measures  of  Performance 


Objective 

Objective  Description 

Comment 

1 

Quality  Systems 

Type  of  quality  system 

2 

Requirements  Definition 

The  degree  of  specific  requirements  communicated  for 
product  and  process 

3 

Design:  Product  /  Process 
&  Risk 

Measurable  Design  Activities  &  Risk  measurement 

4 

Product  /  Process 
Qualification 

From  the  requirements,  key  measurable  performance  of 
product  and  process 

5 

Product  /  Process  Quality 
Metrics 

Measurable  items  of  capability  for  product  and  process 

6 

Customer  Satisfaction 

Measurable  feedback  and  status  of  customer  satisfaction 
factors 
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c 


*  Function 

Technical  Knowledge 


Problem  Statement 

Descriptive:  Poor  Outcomes 
Normative:  Intended  Outcomes 

E 


*  Function 

Design  Stability 

Function 

Manufacturing  Knowledge 


Function 

Evaluation  Objectives 


Objective  2 

Requirements  Definition 


Objective  4 

Product  &  Process  Qualification 


Objective  6 
Customer  Satisfaction 


Objective  1 
Quality  Systems 


Objective  3 

Design  -  Product  /  Process  &  Risk 


Objective  5 

Process  &  Product  Metrics 


Figure  16.  SEDP  Value  Stream — Key  Study  Objectives 


The  modeling  of  weighting  factors  was  developed  for  the  six  high  level  objectives 
in  the  scoring  system.  Each  objective  is  weighted  by  the  mapping  of  the  37  individual 
MOPs  and  their  association  with  11  normative  attributes.  See  Appendix  K  MOP 
Rationale  and  Weighted  Factors.  An  aggregate  total  score  sums  across  each  of  the  six 
objective  categories.  If  a  given  alternative  under  study  did  not  use  a  listed  best  practice 
attribute,  then  it  would  not  receive  a  counting  value  in  the  aggregate  of  the  objective 
measure.  An  example  of  a  notional  raw  data  matrix  worksheet  shows  how  scores  roll  into 
the  raw  data  matrix  for  each  alternative  (Table  16).  By  applying  these  weighting  factors 
to  the  countable  score  of  the  assessment  criteria,  then  a  selection  from  the  alternatives 
may  emerge  as  a  preferred  best  practice  overall. 
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Table  16.  .Notional  AOA  Excel  Tool  Raw  Data  Matrix  Development. 


Objective  3 

Design  -  Product  /  Process  &  Risk 


Objective  4 

Product  &  Process  Qualification 


Objective  Evidence  1 .1 

QMS  ISO  9000  AS9100  ISO/TS  16949w/  Certification 


/ - \ 

Objective  Evidence  1 .2 

APQP  Manufacturing  Development  Guidelines 


Objective  Evidence  1 .3 

PPAP  Followed  -  Pre-Production 
v _ / 


Objective  5 

X 

Process  &  Product  Metrics 

V 

Objective  6 
Customer  Satisfaction 
_ 


Figure  17.  Objective  1  -  Value  Hierarchy — Quality  Systems. 
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Objective  Evidence  2.1 

Design  Responsible  Activities 

Objective  Evidence  2.2 

Build  to  Print  Activities 

f  A 

Objective  Evidence  2.3 

Make/Buy  Activities 

Objective  Evidence  2.4 

Product  Qualification  -  Performance 

r  a 

Objective  Evidence  2.5 

Measurement  System  Compliant 

S. _ j 

f  A 

Objective  Evidence  2.6 

Process  Validation  (Stability  /  Control) 

Figure  18.  Objective  2  -  Value  Hierarchy — Requirements  Definition. 


Figure  19.  Objective  3  -  Value  Hierarchy — Design  -  Product  /  Process  Risk. 
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Process  &  Product  Metrics 
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Objective  Evidence  4.1 

Measurement  System  Evaluation 

Objective  Evidence  4.2 

Product  Qualification,  ATP  &  FAIT 

r  's 

Objective  Evidence  4.3 

Process  Verification  -  Attributes  &  Variables  -  SPC 

r  a 

Objective  Evidence  4.4 

Supply  Network  -  APQP 

Objective  Evidence  4.5 

Supply  Network  -  FMEA 

Objective  Evidence  4.6 

Supply  Network  -  MSA 

r  > 

Objective  Evidence  4.7 

Supply  Network  -  SPC 

Objective  Evidence  4.8 

Supply  Network -PPAP 

Figure  20.  Objective  4  -  Value  Hierarchy — Product  and  Process  Qualification. 


Objective  Evidence  5.1 

Field  Failures  -  FRACAS 

Objective  Evidence  5.2 

Process  Failures  -  Non-Conformance  FRACAS 

Objective  Evidence  5.3 

Yield  /  Scrap  Metrics  -  Targets  -  Achievements 

r  ^ 

Objective  Evidence  5.4 

Lean  /  Six  Sigma  -  DIMAC 

Objective  Evidence  5.5 

Qualification  -  Test 

Objective  Evidence  5.6 

Field  Performance  -  Warranty 

Objective  Evidence  5.7 

Things  Gone  Wrong  /  Right 

Figure  21.  Objective  5  -  Value  Hierarchy — Product  and  Process  Metrics. 
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Figure  22.  Objective  6  -  Value  Hierarchy — Satisfaction  and  Economics. 


D.  AOA  DETAILS 

The  AOA  with  its  objectives  and  defined  MOPS  allows  a  comparative  scoring 
based  upon  an  alternative’s  use  of  individual  best  practices.  If  the  alternative  production 
approval  process  utilizes  a  lower-level  MOP  best  practice  then  a  countable  score  would 
be  awarded  (Table  17). 


Table  17.  SEDP  Detailed  Lower- Level  Measures  of  Performance 


Manufacturing  Knowledge  Gaps  and  Measures  of  Performance 

Objective  1.  Quality  Systems 

Sub-Function  Measures  of  Performance 

QMS,  APQP  &  PPAP  w/  3rd  Party 
Compliance 

1.  QMS  -  ISO  9000/  AS9100  /  ISO/TS  16949 

2.  APQP  Manufacturing  Development  Guidelines 

3.  PPAP  Followed  -  Pre-Production 

Objective  2.  Requirements 

Sub-Function  Measures  of  Performance 

Product  &  Process  w/  PPAP 

Warrant 

1 .  Common  Training  -  infrastructure 

2.  Development  Activities  -  APQP 

3.  Warrant  Approach  -  PPAP 

4.  Product  /Process  -  Performance  -  DFMEA/PFMEA 

5.  Measurement  System  Analysis 

6.  Process  Validation  (Stability  &  Control  -  SPC) 

Objective  3.  Design  &  Risk 

Sub-Function  Measures  of  Performance 
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Manufacturing  Knowledge  Gaps  and  Measures  of  Performance 


Product  Definition,  Maturity 

1 .  PDR,  CDR  %  Drawings  Completed 

2.  Drawing  Controls  CM 

3.  DOEs  -  Critical  Characteristics  Defined 

Process  Definition  -  Stability  and 
Control 

4.  Not  Stable:  Pp  Ppk  >=  1.33  -  LRIP 

5.  Stable:  Cpk  >=  1.67  -  FRP 

Risks  Mitigated  &  Issues  Resolved 

6.  Product  &  Process  Maturation 

7.  Programmatic  Corrective  Action  System 

8.  DFMEA/PFMEA 

Objective  4.  Qualification 

Sub-Function  Measures  of  Performance 

Product  &  Process  Qualification 
Integration 

Supply  Network 

1 .  Measurement  System  Evaluation 

2.  Product  Qualification  -  Qualification,  Acceptance  Test 
Procedure  &  First  Article  Inspection  and  Test 

3.  Process  Verification  -  Attributes  &  Variables 

4.  Supply  Network  APQP 

5.  Supply  Network  DFMEA/PFMEA 

6.  Supply  Network  MSA 

7.  Supply  Network  SPC 

8.  Supply  Network  PPAP 

Objective  5.  Metrics 

Sub -Function  Measures  of  Performance 

Corrective  Action  System 
Non-Conformance  /  Yield  /Scrap 

1 .  Field  Failures  -  FRACAS 

2.  Process  Failures  -  Non-Conformance  FRACAS 

3.  Yield  /  Scrap  Metrics  -  Targets  -  Achievements 

4.  Fean  /  Six  Sigma  Results  -  DIMAC 

Reliability  /  Durability 

5.  Qualification  -  Test 

6.  Field  Performance  -  Warranty 

7.  Things  Gone  Wrong  /  Right 

Objective  6.  Customer  Satisfaction 

Sub-Function  Measures  of  Performance 

Quality  Economics 

PAF  Model  of  Quality  Improvement 

1.  Metrics  Established  -  Enterprise 

2.  Metrics  Established  -  Focal 

3.  PAF  Managed 

Business  Economics 

4.  Sales  Performance 

5.  Market  Share  Performance 

In  the  SEDP  approach,  the  AOA  has  a  focus  centered  on  high-level  objectives  and 
deriving  lower-level  functional  requirements.  These  lower- level  requirements  challenge 
the  solution  space.  In  this  study,  the  AOA  identified  functional  requirements  by  first 
creating  a  value  stream  functional  analysis.  The  input  of  the  best  practice  tally  goes  into 
the  tabular  form  to  capture  scores  by  alternative  by  objective.  The  scoring  possible,  for 
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each  objective,  is  just  the  number  of  lower-level  countable  MOPs  under  that  objective 
(See  Table  18  through  Table  20). 


Table  18.  SEDP  AOA  Results:  Objective  1  and  2 — Raw  Evidence  Tally. 


MOPs  Performed  by  Alternative 

Evaluation  Category:  Manufacturing  Knowledge  Gaps  and  Associated 

Measures  of  Performance  Category  by  Alternative 

DOD 

Weapons 

FDA 

Medical 

Devices 

FAA 

Aviation 

Auto 

Vehicles 

1.  Quality  Systems 

Objective  MOP  1  Roll  up — > 

i 

3 

3 

3 

QMS,  APQP  &  PPAP  w/3rd  Party  Compliance 

1.  QMS  -  ISO  9000/  AS9100  /  ISO/TS  16949 

2.  APQP  Manufacturing  Development  Guidelines 

3.  PPAP  Followed  -  Pre-Production 

i 

1,2,3 

1,2.3 

1,2,3 

2.  Requirements  Understood 

Objective  MOP  2  Roll  up  — > 

3 

4 

4 

6 

Product  &  Process  w/  PPAP  Warrant 

1.  Common  Training  -  infrastructure 

2.  Development  Activities  -  APQP 

3.  Warrant  Approach  -  PPAP 

4.  Product  /Process  -  Performance  -  FMEA 

5.  Measurement  System  Compliant  (MSA) 

6.  Process  Validation  (Stability  &  Control  -  SPC) 

2,5 

2, 4, 5, 6 

2, 4, 5, 6 

1,2, 3, 4, 5, 6 

Table  19.  SEDP  AOA  Results:  Objective  3  and —  Raw  Evidence  Tally. 


MOPs  Performed  by  Alternative 

Evaluation  Category:  Manufacturing  Knowledge  Gaps  and  Associated 
Measures  of  Performance  Category  by  Alternative 

DOD 

Weapo 

ns 

FDA 

Medical 

Devices 

FAA 

Aviation 

Auto 

Vehicles 

1.  Quality  Systems 

Objective  MOP  1  Roll  up - > 

1 

3 

3 

3 

QMS,  APQP  &  PPAP  w/  3rd  Party 
Compliance 

1.  QMS  -  ISO  9000/  AS9100  /  ISO/TS  16949 

2.  APQP  Manufacturing  Development  Guidelines 

3.  PPAP  Followed  -  Pre-Production 

1 

1,2,3 

1,2.3 

1,2,3 

2.  Requirements  Understood 

Objective  MOP  2  Roll  up - > 

3 

4 

4 

6 

Product  &  Process  w/  PPAP  Warrant 

1.  Common  Training  -  infrastructure 

2.  Development  Activities  -  APQP 

3.  Warrant  Approach  -  PPAP 

4.  Product  /Process  -  Performance  -  FMEA 

5.  Measurement  System  Compliant  (MSA) 

6.  Process  Validation  (Stability  &  Control  -  SPC) 

2,5 

2, 4,5, 6 

2, 4,5, 6 

1,2, 3, 4,5, 6 

3.  Design-Product/Process  &  Risk 

Objective  MOP  3  Roll  up - > 

4 

4 

4 

7 

Design  Definition,  Maturity 

1.  PDR,  CDR  %  Drawings  Completed 

2.  Drawing  Controls  CM 

3.  DOEs  -  Critical  Characteristics  Defined 

1,2 

1,2 

1,2 

1,2,3 

Process  Definition  -  Stability  and  Control 

4.  Not  Stable:  Pp  Ppk  >=  1.33  -  LRIP 

5.  Stable:  Cp  Cpk  >=  1.67  -  FRP 

0 

0 

0 

4,5 

Risks  Mitigated  &  Issues  Resolved 

6.  Product  &  Process  Maturation 

7.  Programmatic  Corrective  Action  System 

8.  Design  FMEA  &  PFMEA 

6,7 

6,7 

6,7 

7,8 

4.  Product  &  Process  Qualification 

Objective  MOP  4  Roll  up - > 

4 

4 

5 

8 
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Table  20.  SEDP  AOA  Results:  Objective  5  and  6  -  Raw  Evidence  Tally. 


MOPs  Performed  by  Alternative 

Evaluation  Category:  Manufacturing  Knowledge  Gaps  and  Associated 

Measures  of  Performance  Category  by  Alternative 

DOD 

Weapons 

FDA 

Medical 

Devices 

FAA 

Aviation 

Auto 

Vehicles 

5.  Process  &  Product  /  Metrics 

Objective  MOP  5  Roll  up  — > 

3 

5 

5 

7 

Corrective  Action  System 

Non-Conformance /Yield  /Scrap 

1.  Field  Failures  -  FRACAS 

2.  Process  Failures  -  Non-Conformance  FRACAS 

3.  Yield  /Scrap  Metrics  -  Targets- Achievements 

4.  Lean  /  Six  Sigma  Results  -  DIMAC 

1,2 

1,2,3 

1,2,3 

1,2, 3, 4 

Reliability  /  Durability 

5.  Qualification -Test 

6.  Field  Performance -Warranty 

7.  Things  Gone  Wrong /Right 

5 

5,6 

5,6 

5,6,7 

6.  -  Customer  Satisfaction 

Objective  MOP  6  Roll  up  — > 

2 

4 

4 

4 

Quality  Economics 

PAF  Model  of  Quality  Improvement 

1.  Metrics  Established  -  Enterprise 

2.  Metrics  Established  -  Local 

3.  PAF  Managed 

1,2 

1,2 

1,2 

1,2 

Business  Economics 

4.  Sales  Performance 

5.  Market  Share  Performance 

0 

4,5 

4,5 

4,5 

The  weighting  factors  are  derived  by  the  number  of  best  practice  attributes 
assigned  to  each  of  the  six  objectives.  The  score  would  just  be  the  countable  scored 
points  times  the  weighting  factor.  The  sum  of  the  aligned  weighting  factors  times  the 
objective  aggregate  points  for  each  alternative  is  the  accumulated  score  for  an  alternative 
under  study.  The  largest  score  would  be  the  preferred  solution. 

An  individual  score  was  the  result  of  counting  the  MOPs  by  alternative  then 
rolling  up  the  tallies  as  points.  These  tallies  entered  into  the  AOA  Scoring  matrix.  The 
Microsoft  Excel  tool  then  multiplied  the  weighting  factor  against  the  points  accumulated 
for  each  objective  and  for  each  alternative.  The  summation  of  the  objective  scores  would 
be  a  total  for  the  alternative.  The  preferred  solution  after  scoring  would  be  the  alternative 
with  the  largest  score.  See  Table  21.  The  study  AOA  fully  assessed  shows  the  results  that 
differentiate  upon  the  preferred  solution.  The  study  points  to  the  automotive  production 
approval  process  with  a  score  of  127.  The  next  closest  competing  alternative  achieved  a 
score  of  89. 
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The  results  for  the  more  prescriptive  quality  system,  or  QMS  type,  was 
significance  given  the  finding  of  the  AOA  results  involving  the  DOD,  FAA,  FDA  and 
automotive  industry.  The  production  approval  processes  showing  that  the  automotive 
industry  approach  was  superior  to  all  others  for  cost,  schedule  and  performance  over 
time.  This  analysis  is  one  factor  in  determining  the  desirability  of  a  new  solution  space 
for  DOD  acquisition  practices.  The  APQP/PPAP  process  in  the  automotive  industry 
demonstrated  that  there  is  a  method  in  non-DOD  industries  that  would  satisfy  the 
deficiencies  pointed  out  by  the  GAO:  1)  lack  of  standard  and  systematic  methods  and  2) 
lack  of  knowledge  in  contractor  demonstration  of  production  readiness  prior  to 
production  commitment. 


Table  21.  SEDP  AOA  Results:  Scored  /  Weighted  and  Preferred  Solution 


Value  Hierarchy 

Alternatives  -  Production  Part  Approval  Processes 

Points 

Available 

Weighting 

Function  3:  Manufacturing 
(Measures  of  Performance) 

DOD 

Points  Score 

FDA 

Points  Score 

FAA 

Points  Score 

Auto 

Points  Score 

Obj  1 

Evaluation  3.1  Quality  Systems 

1 

7 

3 

21 

3 

21 

3 

21 

3 

7 

Obj  2 

Evaluation  3.2  Requirements 
Understood 

Obj  3 

Evaluation  3.3  Design- 
Product/Process  &  Risk 

Obj  4 

Evaluation  3.4  Product  &  Process 
Qualification 

Obj  5 

Evaluation  3.5  Process  &  Product 
/  Metrics 

Obj  6 

Evaluation  3.6  -  Customer 

Satisfaction 

3 

9 

4 

12 

4 

12 

6 

18 

6 

3 

4 

8 

4 

8 

5 

10 

7 

14 

8 

2 

4 

24 

4 

24 

4 

24 

8 

48 

4 

6 

3 

6 

5 

10 

5 

10 

7 

14 

7 

2 

2 

6 

4 

12 

4 

12 

4 

12 

5 

3 

127 


Selection  Based  on 
Highest  Score 


Selected 


This  research  used  the  findings  from  the  various  commercial  quality  standards, 
the  literature  review  and  personal  experience  as  a  subject  matter  expert  to  develop  the 
study  assessment  criteria.  Each  criterion  defined  needed  to  have  the  qualities  as  having  a 
sustained  and  widespread  use  throughout  its  constituent  industrial  sector. 
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The  application  of  these  criteria  differentiated  among  candidates  that  one 
alternative  practice  would  be  most  effective.  A  best  practice  improvement  for  DOD 
application  would  close  the  knowledge  gap  as  reported  by  the  GAO  concerning 
manufacturing  practices.  The  inference  is  that  an  improved  process  that  has  proven  to 
achieve  better  production  outcomes  for  the  automotive  sector  would  similarly  improve 
poor  production  outcomes  in  DOD  acquisition.  This  assessment  approach  is  part  of  a 
method  for  a  comparison  of  perspective  best  practices  as  discussed  separately  by  Gardner 
with  his  process  focused  organizational  approach  and  Gibson  with  respect  to  the  SEDP 
process  (Gibson,  Scherer  and  Gibson  2007;  Gardner  2004). 
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V.  FINDINGS 


A.  SUMMARY 

Acquisition  costs  and  fleet  readiness  have  been  affected  adversely  by  poor 
production  outcomes  related  to  a  lack  of  manufacturing  knowledge  at  MS  C.  An 
examination  of  DOD  and  non-DOD  production  approval  processes  identified  an 
opportunity  to  improve  affordability  and  military  readiness  by  improving  the  DOD’s 
approach  to  weapon  systems  production  approval. 

In  this  thesis,  an  analysis  of  alternatives  examined  a  family  of  commercial  best 
practices  used  by  non-DOD  industries  that  followed  a  more  disciplined  approach  to 
manufacturing  development  than  DOD  acquisition.  Quality  standards  used  by  the  non- 
DOD  industrial  sectors  validated  manufacturing  development  and  production  capability 
with  better  production  outcomes.  The  prescriptive  QMS  used  by  the  automotive  industry 
was  significantly  better  than  other  alternatives.  The  AO  A  identified  that  the  automotive 
approach  is  a  preferred  solution  enjoying  superior  quality  and  improved  product 
reliability. 

Specifically,  the  application  of  the  American  automotive  production  approval 
process  helped  suppliers  see  a  gain  in  market  share,  a  reduction  in  costs  and  a  doubling  of 
fielded  reliability  as  a  result  of  implementing  a  unified  and  prescriptive  QMS. 
Automotive  development  required  a  more  disciplined  approach  to  product  and  process 
realization  than  DOD  acquisition  practices.  The  prescriptive  QMS  of  the  automotive 
industry  used  a  knowledge -based  production  approval  process  with  a  production 
submission  warrant  leading  to  production  approval.  The  degree  of  improvement  was 
evident  by  findings  reported  in  commercially  conducted  surveys.  A  listing  of  supporting 
findings  from  this  research  includes: 

•  Department  of  Defense  weapon  systems  production  outcomes  fall  short  of 
program  goals  with  up  to  34%  of  poor  production  outcomes  coming  from 
manufacturing  and  quality  issues. 
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•  Department  of  Defense  acquisition  policy  provides  minimal  direction  with 
respect  to  manufacturing  development  and  verification  of  a  production 
capability  at  MS  C. 

•  Non-Department  of  Defense  a  production  part  approval  processes  helped 
identify  industrial  best  practices  that  demonstrated  that  a  prescriptive 
QMS  (QS-9000)  would  improve  quality  and  reliability  better  than  non- 
prescriptive  QMS  (ISO-9000). 

•  Manufacturers  experienced  significant  improvements  in  quality  and 
reliability  based  on  the  applying  the  automotive  QMS.  This  QMS  guided 
manufacturing  development  with  a  prescriptive  manufacturing 
development  process  -  APQP  and  then  required  a  demonstration  of 
production  capability  prior  to  production  start  using  a  certification  process 
-  PPAP. 

•  The  aerospace  standards  committee  of  the  SAE  realized  in  2014  that  the 
automotive  approach  would  improve  quality  outcomes  for  the  DOD  and 
aerospace  industry  and  added  to  the  AS-9000  family  of  standards  with  the 
release  of  AS-9145  APQP  and  PPAP  on  November  8,  2016. 

•  Department  of  Defense  technical  authority  structure  could  invoke  the  AS- 
9145  as  a  certification  process  to  warrant  production  readiness  compliance 
similar  to  non-DOD  industries. 

The  DOD  is  able  to  invoke  these  or  related  standards  through  the  applicable  Code 
of  Federal  Regulations.  See  Title  48,  sec.  46,  higher  QMS  requirements  (Code  of  Federal 
Regulations,  title  48,  sec.  46.202.4). 

B.  CONCLUSIONS 

This  study  was  significant  in  that  it  addressed  a  gap  in  the  research  with  respect  to 
manufacturing  causality  of  poor  production  outcomes  in  DOD  acquisition.  The  findings 
are  significant  identifying  critical  processes  behind  a  more  disciplined  manufacturing 
development  and  demonstration  of  production  capability  in  non-DOD  industries.  Non- 
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DOD  industrial  sectors  required  a  certified  demonstration  of  production  readiness  prior  to 
production  commitment  and  addressed  the  DOD’s  poor  production  outcome  causal 
factors.  Non-DOD  production  approval  processes  studied  used  a  more  prescriptive  QMS 
where  compelling  evidence  of  improvement  potential  was  confirmed  by  commercial 
surveys  comparing  industry  QMS.  Industries  that  followed  quality  standards  that  address 
manufacturing  development  using  an  advanced  product  quality  planning  process  with  a 
warranted  production  part  approval  process  preformed  significantly  better  as  contrasted 
with  non-prescriptive  QMS  as  found  in  the  AS-9100  and  ISO-9000  standards. 

C.  RECOMMENDATIONS 

The  automotive  industry  has  relied  upon  the  APQP/PPAP  as  part  of  a  disciplined 
SE  application  to  the  manufacturing  development  process  and  has  been  highly  effective 
for  decades.  The  PPAP  warrant  process,  as  required  by  the  automotive  OEMs,  is  flowed 
down  to  the  lowest-level  supplier  showing  consistency  of  process  and  production  results. 
The  DOD  should  adopt  the  APQP/PPAP  process  to  support  a  disciplined  manufacturing 
development  toward  an  earlier  demonstrated  production  capability  at  the  end  of  EMD. 
This  would  likely  bring  the  pre-FRP  PCA  into  EMD  as  a  process  improvement  in  DOD 
acquisition. 

The  DOD  should  continuing  developing  workforce  enhancement  in  their 
competency  aligned  organizations  with  respect  to  manufacturing  expertise.  Additional 
workforce  enhancement  could  be  realized  by  using  a  third  party  organization  such  as  the 
AIAG  to  train  and  enable  all  suppliers  across  the  industrial  base  adjunct  to  being 
compliant  to  a  required  prescriptive  QMS.  The  DOD  should  update  the  manufacturing 
related  policy  documents  needed  to  implement  the  preferred  prescriptive  QMS  as  a 
requirement  for  the  DOD  supply  base.  Updated  acquisition  guidance  should  be  developed 
in  support  of  the  APQP  and  PPAP.  The  DOD  application  of  AS-9100  revision  D  should 
be  required  with  its  related  AS-9145  standard  that  calls  for  APQP  and  PPAP. 

This  study  supports  the  standardization  of  production  readiness  demonstration  for 
use  in  DOD  acquisition.  Prime  contractors  would  engage  their  supply  networks  to  apply 
the  prescriptive  QMS  as  a  standard  practice.  Program  management  responsibility  would 
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change  from  a  reporting  of  manufacturing  risk  to  a  reporting  of  process  capability  in  a 
knowledge-based  approach.  Technical  authorities  should  identify  production  capability  as 
a  “critical  performance  parameter”  that  would  be  required  to  establish  a  certification 
process  in  acquisition  and  appropriately  update  the  JCIDS  to  include  guidance  for  a 
PPAP  approach.  Here,  the  TA  serving  as  a  chief  engineer  (CHENG)  within  a  program 
would  represent  process  capability  demonstration  under  a  warrant  process.  Finally,  a  pilot 
program  on  large  scale  would  better  assess  the  impacts  and  benefits  in  a  DOD  application 
of  an  APQP  /  PPAP. 

The  list  of  DOD  acquisition  structural  enablers  for  APQP/PPAP  implementation 
comes  from  an  operational  approach  to  include: 

•  Engage  Stakeholders:  (OSD[AT&L],  DDRE,  MANTECH,  NDIA, 
COCOMs,  AIAG,  SAE,  ISO,  ASQ  )  Identify  training  roles  and 
responsibilities. 

•  Synthesize  Comparative  Processes:  DOD/SETR  process  with  PCA  to  the 
ISO/TS  16949/AS9145  and  the  requirements  of  the  PPAP  with  Technical 
Warrant  /  Certification  -  through  Supply  Management;  SE  Manufacturing 
and  Quality. 

•  Engage  the  Policy  Document  Owners:  Define  Roles  and  Responsibilities  - 
Within  DOD  policy,  instructions  and  COCOMs  (e.g.,  Naval  Air  Systems 
Command  /  Naval  Sea  System  Command,  Air  Force  and  Army). 

•  Engage  a  third  party  expertise  to  enable  the  supply  network,  such  as  the 
role  found  in  the  AIAG  automotive  consortium  with  respect  to  training 
and  implementation  of  the  APQP/PPAP. 

•  Consider  Acquisition  Reform  in  contracting  for  DOD  Weapon’s  System 
programs  where  the  role  of  enhanced  QMS  is  standard  and  by  exception  if 
not  used. 

•  Consider  identifying  pilot  contracts  for  implementation  using  AS9100  C 
Clause  3.2  for  APQP  and  PPAP  customer  requirements  according  to  FAR 
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46.202-4  (see  AS-9100:2016  with  attachment  of  AS9145  -  -  November 
2016  release)  to  impact  the  entire  supply  network  of  a  prime  contractor. 

•  Strengthen  the  workforce  competency,  experience,  and  requirements  to 
address  what  is  the  current  state  of  DOD  acquisition  practices  covering 
poor  production  outcomes  impact  related  to  manufacturing  and  a  transition 
to  the  use  of  AS  9 145. 

•  Apply  a  warrant  process  for  production  readiness  as  described  in  the  SAE 
standard,  AS-9145  APQP  and  PPAP  with  its  PSW  requirement.  The 
purpose  of  the  warrant  leverages  the  benefits  of  this  research  as  a  best 
practice  (Appendix  E  -  DOD  Technical  Warrants). 

The  DOD  should  examine  the  APQP/PPAP  approach  from  any  wide  spread  pilot 
program  within  the  DOD  acquisition  process  as  a  pathfinder  for  deploying  a  knowledge- 
based  acquisition  for  production  approval.  There  is  a  presumption  that  this  best  practice 
from  the  automotive  industry  is  now  accepted  by  the  SAE  through  the  new  AS-9145 
standard  as  modeled  after  the  automotive  best  practices  (SAE  2015). 

DOD  would  develop  results  using  a  knowledge-based  production  readiness 
approach  while  lowering  the  current  risk-based  manufacturing  readiness  practices. 
Results  of  production  readiness  demonstration  would  inform  the  MDA  acquisition 
executives  with  confidence  in  an  actual  system  at  MS  C. 

D.  FUTURE  STUDY 

Future  study  could  include  other  items  that  would  support  this  thesis  that  could 
not  be  employed  due  to  lack  of  time  or  funding.  The  following  list  includes  potential 
relevant  ideas  which  do  not  fall  within  the  scope  of  this  thesis. 

•  Develop  detailed  policy  change  documents  covering  the  DAS  including: 

•  JCIDS;  DAG;  SYSCOM  Instructions;  MIL-HDBK-896  with  AS9145 

•  Interview  the  SAE  G-14  members  that  developed  the  AS9145  APQP  / 
PPAP  and  their  constituents 


99 


•  Develop  Interview  Questions;  Identify  Subjects;  Submit  to  IRB  for 
review;  Conduct  Interview;  Analyze  Results 

•  Develop  an  operational  implementation  strategy  and  templates  for  all 
SYSCOM  implementation:  Contract  Language;  CDRLS;  DIDs;  SOWs; 
TEMP;  CDP 

•  Define  Manufacturing  Requirements  for  DOD  Specification  application 
Cpk;  Ppk;  Yield;  PSW  format  and  content 

•  Develop  strategy  for  Technical  Warrant  issuance  for  PSW  in  DOD 
contracting 


100 


APPENDIX  A.  PERRY  MEMO 


The  William  Perry  memo  is  presented  in  its  entirety  and  is  provided  verbatim. 


THE  SECRETARY  OF  DEFENSE 
WASHINGTON,  DC  20301-1000 
29  Jun  94 


MEMORANDUM  FOR  SECRETARIES  OF  THE  MILITARY  DEPARTMENTS 
CHAIRMAN  OF  THE  JOINT  CHIEFS  OF  STAFF 
UNDER  SECRETARIES  OF  DEFENSE 
COMPTROLLER 

ASSISTANT  SECRETARY  OF  DEFENSE  (COMMAND,  CONTROL, 

COMMUNICATIONS,  AND  INTELLIGENCE) 

GENERAL  COUNSEL 
INSPECTOR  GENERAL 

DIRECTOR  OF  OPERATIONAL  TEST  AND  EVALUATION 
DIRECTORS  OF  THE  DEFENSE  AGENCIES 
COMMANDER-IN-CHIEF,  U.S.  SPECIAL  OPERATIONS 
COMMAND 

SUBJECT:  Specifications  &  Standards  -  A  New  Way  of  Doing  Business 

To  meet  future  needs,  the  DOD  must  increase  access  to  commercial  state-of-the-art 
technology  and  must  facilitate  the  adoption  by  its  suppliers  of  business  processes 
characteristic  of  world  class  suppliers.  In  addition,  integration  of  commercial  and  military 
development  and  manufacturing  facilitates  the  development  of  dual-use  processes  and 
products  and  contributes  to  an  expanded  industrial  base  that  is  capable  of  meeting  defense 
needs  at  lower  costs. 


I  have  repeatedly  stated  that  moving  to  greater  use  of  performance  and  commercial 
specifications  and  standards  is  one  of  the  most  important  actions  that  DOD  must  take  to 
ensure  we  are  able  to  meet  our  military,  economic,  and  policy  objectives  in  the  future. 
Moreover,  the  Vice  President’s  National  Performance  Review  recommends  that  agencies 
avoid  government-unique  requirements  and  rely  more  on  the  commercial  marketplace. 

To  accomplish  this  objective,  the  Deputy  Under  Secretary  of  Defense  (Acquisition 
Reform)  chartered  a  Process  Action  Team  to  develop  a  strategy  and  a  specific  plan  of 
action  to  decrease  reliance,  to  the  maximum  extent  practicable,  on  military  specifications 
and  standards.  The  Process  Action  Team  report,  “Blueprint  for  Change,”  identifies  the 
tasks  necessary  to  achieve  this  objective.  I  wholeheartedly  accept  the  Team’s  report  and 
approve  the  report’s  primary  recommendation  to  use  performance  and  commercial 
specifications  and  standards  in  lieu  of  military  specifications  and  standards,  unless  no 
practical  alternative  exists  to  meet  the  user’s  needs.  I  also  accept  the  report  of  the  Industry 
Review  Panel  on  Specifications  and  Standards  and  direct  the  Under  Secretary  of  Defense 
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(Acquisition  and  Technology)  to  appropriately  implement  the  Panel’s  recommendations. 

I  direct  the  addressees  to  take  immediate  action  to  implement  the  Team’s 
recommendations  and  assign  the  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  overall  implementation  responsibility.  I  direct  the  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  to  immediately  arrange  for  reprogramming  the 
funds  needed  in  FY94  and  FY95  to  efficiently  implement  the  recommendations.  I  direct 
the  Secretaries  of  the  Military  Departments  and  the  Directors  of  the  Defense  Agencies  to 
program  funding  for  FY96  and  beyond  in  accordance  with  the  Defense  Planning 
Guidance. 


Policy  Changes 

Listed  below  are  a  number  of  the  most  critical  changes  to  current  policy  that  are  needed  to 
implement  the  Process  Action  Team’s  recommendations.  These  changes  are  effective 
immediately.  However,  it  is  not  my  intent  to  disrupt  on-going  solicitations  or  contract 
negotiations.  Therefore,  the  Component  Acquisition  Executive  (as  defined  in  Part  15  of 
DOD  Instruction  5000.2),  or  a  designee,  may  waive  the  implementation  of  these  changes 
for  on-going  solicitations  or  contracts  during  the  next  180  days  following  the  date  of  this 
memorandum.  The  Under  Secretary  of  Defense  (Acquisition  and  Technology)  shall 
implement  these  policy  changes  in  DOD  Instruction  5000.2,  the  Defense  Federal 
Acquisition  Regulation  Supplement  (DFARS),  and  any  other  instructions,  manuals, 
regulations,  or  policy  documents,  as  appropriate. 

Military  Specifications  and  Standards:  Performance  specifications  shall  be  used  when 
purchasing  new  systems,  major  modifications,  upgrades  to  current  systems,  and  non- 
developmental  and  commercial  items,  for  programs  in  any  acquisition  category.  If  it  is  not 
practicable  to  use  a  performance  specification,  a  non-govemment  standard  shall  be  used. 
Since  there  will  be  cases  when  military  specifications  are  needed  to  define  an  exact  design 
solution  because  there  is  no  acceptable  non-govemmental  standard  or  because  the  use  of  a 
performance  specification  or  non-government  standard  is  not  cost  effective,  the  use  of 
military  specifications  and  standards  is  authorized  as  a  last  resort,  with  an  appropriate 
waiver. 

Waivers  for  the  use  of  military  specifications  and  standards  must  be  approved  by  the 
Milestone  Decision  Authority  (as  defined  in  Part  2  of  DOD  Instruction  5000.2).  In  the  case 
of  acquisition  category  ID  programs,  waivers  may  be  granted  by  the  Component 
Acquisition  Executive,  or  a  designee.  The  Director,  Naval  Nuclear  Propulsion  shall 
determine  the  specifications  and  standards  to  be  used  for  naval  nuclear  propulsion  plants  in 
accordance  with  Pub.  L.  98-525  (42  U.S.C.  §7158  note).  Waivers  for  reprocurement  of 
items  already  in  the  inventory  are  not  required.  Waivers  may  be  made  on  a  “class”  or  items 
basis  for  a  period  of  time  not  to  exceed  two  years. 

Innovative  Contract  Management:  The  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  shall  develop,  within  60  days  of  the  date  of  this  memorandum,  Defense 
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Federal  Acquisition  Regulation  Supplement  (DFARS)  language  to  encourage  contractors 
to  propose  non-government  standards  and  industry-wide  practices  that  meet  the  intent  of 
the  military  specifications  and  standards.  The  Under  Secretary  will  make  this  language 
effective  180  days  after  the  date  of  this  memorandum.  This  language  will  be  developed  for 
inclusion  in  both  requests  for  proposal  and  in  on-going  contracts.  These  standards  and 
practices  shall  be  considered  as  alternatives  to  those  military  specifications  and  standards 
cited  in  all  new  contracts  expected  to  have  a  value  of  $100,000  or  more,  and  in  existing 
contracts  of  $500,000  or  more  having  a  substantial  contract  effort  remaining  to  be 
performed. 

Pending  completion  of  the  language,  I  encourage  the  Secretaries  of  the  Military 
Departments  and  the  Directors  of  the  Defense  Agencies  to  exercise  their  existing  authority 
to  use  solicitation  and  contract  clause  language  such  as  the  language  proposed  in  the 
Process  Action  Team’s  report.  Government  contracting  officers  shall  expedite  the 
processing  of  proposed  alternatives  to  military  specifications  and  standards  and  are 
encouraged  to  use  the  Value  Engineering  no-cost  settlement  method  (permitted  by  FAR 
48.104-3)  in  existing  contracts. 

Program  Use  of  Specifications  and  Standards:  Use  of  specifications  and  standards  listed 
in  DOD  Instruction  5000.2  is  not  mandatory  for  Program  Managers.  These  specifications 
and  standards  are  tools  available  to  the  Program  Manager,  who  shall  view  them  as 
guidance,  as  stated  in  Section  6-Q  of  DOD  Instruction  5000.2. 

Tiering  of  Specification  and  Standards:  During  production,  those  system  specifications, 
subsystem  specifications  and  equipment/product  specifications  (through  and  including  the 
first-tier  reference  in  the  equipment/product  specifications)  cited  in  the  contract  shall  be 
mandatory  for  use.  Lower  tier  references  will  be  for  guidance  only,  and  will  not  be 
contractually  binding  unless  they  are  directly  cited  in  the  contract.  Specifications  and 
standards  listed  on  engineering  drawings  are  to  be  considered  as  first-tier  references. 
Approval  of  exceptions  to  this  policy  may  only  be  made  by  the  Head  of  the  Departmental 
or  Agency  Standards  Improvement  Office  and  the  Director,  Naval  Nuclear  Propulsion  for 
specifications  and  drawings  used  in  nuclear  propulsion  plants  in  accordance  with  Pub.  L. 
98-525  (42  U.S.C.  §7158  Note). 


New  Directions 


Management  and  Manufacturing  Specifications  and  Standards:  Program  Managers 
shall  use  management  and  manufacturing  specifications  and  standards  for  guidance  only. 
The  Under  Secretary  of  Defense  (Acquisition  and  Technology)  shall  develop  a  plan  for 
canceling  these  specifications  and  standards,  inactivating  them  for  new  designs, 
transferring  the  specifications  and  standards  to  non-govemment  standards,  converting 
them  to  performance-based  specifications,  or  justifying  their  retention  as  military 
specifications  and  standards.  The  plan  shall  begin  with  the  ten  management  and 
manufacturing  standards  identified  in  the  Report  of  the  Industry  Review  Panel  on 
Specifications  and  Standards  and  shall  require  completion  of  the  appropriate  action,  to  the 
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maximum  extent  practicable,  within  two  years. 

Configuration  Control:  To  the  extent  practicable,  the  Government  should  maintain 
configuration  control  of  the  functional  and  performance  requirements  only,  giving- 
contractors  responsibility  for  the  detailed  design. 

Obsolete  Specifications:  The  “Department  of  Defense  Index  of  Specifications  and 
Standards”  and  the  “Acquisition  Management  System  and  Data  Requirements  Control 
List”  contain  outdated  military  specifications  and  standards  and  data  requirements  that 
should  not  be  used  for  new  development  efforts.  The  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  shall  develop  a  procedure  for  identifying  and  removing 
these  obsolete  requirements. 

Use  of  Non-Government  Standards:  I  encourage  the  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  to  form  partnerships  with  industry  associations  to  develop 
non-government  standards  for  replacement  of  military  standards  where  practicable.  The 
Under  Secretary  shall  adopt  and  list  in  the  “Department  of  Defense  Index  of  Specifications 
and  Standards”  (DoDISS)  non-government  standards  currently  being  used  by  DOD.  The 
Under  Secretary  shall  also  establish  teams  to  review  the  federal  supply  classes  and 
standardization  areas  to  identify  candidates  for  conversion  or  replacement. 

Reducing  Oversight:  I  direct  the  Secretaries  of  the  Military  Departments  and  the 
Directors  of  the  Defense  Agencies  to  reduce  direct  Government  oversight  by  substituting 
process  controls  and  non-govemment  standards  in  place  of  development  and/or  production 
testing  and  inspection  and  military-unique  quality  assurance  systems. 

Cultural  Changes 

Challenge  Acquisition  Requirements:  Program  Managers  and  acquisition  decision 
makers  at  all  levels  shall  challenge  requirements  because  the  problem  of  unique  military 
systems  does  not  begin  with  the  standards.  The  problem  is  rooted  in  the  requirements 
determination  phase  of  the  acquisition  cycle. 

Enhance  Pollution  Controls:  The  Secretaries  of  the  Military  Departments  and  the 
Directors  of  the  Defense  Agencies  shall  establish  and  execute  an  aggressive  program  to 
identify  and  reduce  or  eliminate  toxic  pollutants  procured  or  generated  through  the  use  of 
specifications  and  standards. 

Education  and  Training:  The  Under  Secretary  of  Defense  (Acquisition  and  Technology) 
shall  ensure  that  training  and  education  programs  throughout  the  Department  are  revised  to 
incorporate  specifications  and  standards  reform. 

Program  Reviews:  Milestone  Decision  Authority  (MDA)  review  of  programs  at  all  levels 
shall  include  consideration  of  the  extent  streamlining,  both  in  the  contract  and  in  the 
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oversight  process,  is  being  pursued.  The  MDA  (i.e.,  the  Component  Acquisition  Executive 
or  his/her  designee,  for  all  but  AC  AT  ID  programs)  will  be  responsible  for  ensuring  that 
progress  is  being  made  with  respect  to  programs  under  his/her  cognizance. 

Standards  Improvement  Executives:  The  Under  Secretary  the  Secretaries  of  the  Military 
Departments  and  the  Director  of  the  Defense  Logistics  Agency  shall  appoint  Standards 
Improvement  Executives  within  30  days.  The  Standards  Improvement  Executives  shall 
assume  the  responsibilities  of  the  current  Standardization  Executives,  support  those 
carrying  out  acquisition  reform,  direct  implementation  of  the  military  specifications  and 
standards  reform  program,  and  participate  on  the  Defense  Standards  Improvement 
Council.  The  Defense  Standards  Improvement  Council  shall  be  the  primary  coordinating 
body  for  the  specification  and  standards  program  within  the  Department  of  Defense  and 
shall  report  directly  to  the  Assistant  Secretary  of  Defense  (Economic  Security).  The 
Council  shall  coordinate  with  the  Deputy  Under  Secretary  of  Defense  (Acquisition 
Reform)  regarding  specification  and  standards  reform  matters,  and  shall  provide  periodic 
progress  reports  to  the  Acquisition  Reform  Senior  Steering  Group,  who  will  monitor 
overall  implementation  progress. 

Management  Commitment 

This  Process  Action  Team  tackled  one  of  the  most  difficult  issues  we  will  face  in 
reforming  the  acquisition  process.  I  would  like  to  commend  the  team,  composed  of 
representatives  from  all  of  the  Military  Departments  and  appropriate  Defense  Agencies, 
and  its  leader,  Mr.  Harold  Griffin,  for  a  job  well  done.  In  addition,  I  would  like  to  thank 
the  Army,  and  in  particular,  Army  Materiel  Command,  for  its  administrative  support  of  the 
team. 

The  Process  Action  Team’s  report  and  the  policies  contained  in  this  memorandum  are  not 
a  total  solution  to  the  problems  inherent  in  the  use  of  military  specifications  and  standards; 
however,  they  are  a  solid  beginning  that  will  increase  the  use  of  performance  and 
commercial  specifications  and  standards.  Your  leadership  and  good  judgment  will  be 
critical  to  successful  implementation  of  this  reform.  I  encourage  you  and  your  leadership 
teams  to  be  active  participants  in  establishing  the  environment  essential  for  implementing 
this  cultural  change. 

This  memorandum  is  intended  only  to  improve  the  internal  management  of  the  Department 
of  Defense  and  does  not  create  any  right  or  benefit,  substantive  or  procedural,  enforceable 
at  law  or  equity  by  a  party  against  the  Department  of  Defense  or  its  officers  and 
employees. 


/signed/ 
William  J.  Perry 
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APPENDIX  B.  GAO  MEMO 


This  memo  from  the  GAO  describes  the  ethics  and  professional  standards  that 
underpin  their  reports. 


U.S.  GOVERNMENT  ACCOUNTABILITY  OFFICE 


Memorandum 


Date:  June  2,  2015 

To:  GAO’s  Financial  Management  and  Assurance  Team  (FMA) 

CIGIE  Financial  Audit  Community 

From:  James  R.  Dalkin,  Director,  FMA 

Jessica  Boucher,  Eric  Holbrook,  William  Lowrey,  Greg  Ziombra 

Subject:  Professional  Standards  Update  No.  62 

February  2014  through  May  2015 

We  issue  Professional  Standards  Updates  (PSU)  to  alert  staff  of  changes  in  professional 
standards  and  related  guidance.  The  PSUs  highlight  some  key  points  concerning  recent  activity 
at  standard-setting  organizations.  PSUs  are  not  intended  to  provide  comprehensive  information 
on  all  standards.  Those  affected  by  a  new  standard  should  refer  that  standard  for  details. 
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APPENDIX  C.  FAA  MEMO 


This  memo  from  the  FAA  is  an  advisory  in  support  of  the  regulations  regarding 
production  approval. 


AVIATION 

RULEMAKING 

ADVISORY 

COMMITTEE 


Mr.  Nicholas  A.  Sabatini 
Associate  Administrator  for  Regulations  and  Certification 
Federal  Aviation  Administration 
800  Independence  Avenue,  S.W 
Washington,  DC  20591 


Subject:  Transmittal  of  Documents  in  Support  of  ARAC  21  Production  Certification  and 
Parts  Approval  Tasking 

Dear  Mr  Sabatini: 

This  forwards  the  following  documents,  which  have  been  prepared  in  support  of  the  ARAC  21 
Production  Certification  and  Parts  Approval  tasking 

•  Means  of  Compliance  with  Proposed  Quality  System  Requirements,  2-20-02 

•  PAH  Transition  to  New  Quality  System  Requirements;  2-20-02 

•  Quality  System  Guidance;  1 1-26-01 

•  Proposed  AC  on  Standard  and  Commercial  Parts;  3-21-02 

•  ODAR,  3-21-02 

•  PDA  Document,  1 1-26-01 

This  is  the  culmination  of  a  long  effort  to  enhance  the  safety  of  the  production  certification  system 
for  aeronautical  products  and  those  parts  needed  for  continued  operational  safety 

FAA  first  began  this  effort  in  1986.  In  1993,  FAA  tasked  ARAC  21  to  develop  recommendations 
for  proposed  regulations  and  guidance  materials  that  would  accomplish  this  safety  objective.  The 
preponderance  of  ARAC  21  recommendations  on  this  tasking  were  submitted  to  FAA  in  1999  and 


March  26,  2002 
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would  have  enabled  the  FAA  to  proceed  with  the  development  of  an  Notice  of  Proposed 
Rulemaking.  However,  since  an  NPRJV1  has  not  yet  been  developed,  the  ARAC  21,  in  anticipation 
that  an  NPRJVI  would  be  forthcoming,  continues  to  support  the  FAA  with  the  development  of 
additional  recommended  guidance  for  important  aspects  of  this  safety  effort. 

The  industry  and  ARAC  21  respectfully  request  the  FAA  consider  assigning  a  higher  priority  to 
this  safety  rulemaking  effort. 


Very  truly  yours, 


W.  H.  Schultz 
Assistant  ARAC  Chair 
ARAC  2 1  Issues 


Attachments 


Copies: 

John  Hickey,  AIR-1 
Frank  Paskiewicz,  AIR-200 
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Mr.  Bill  Schultz 

Assistant  Chair,  Aircraft  Certification 
Procedures  Issues 
1400  K  Street  NW. 

Washington,  DC  20005 

Dear  Mr.  Schultz: 


This  letter  acknowledges  receipt  of  your  March  26  letter  and  email  message  transmitting 
guidance  recommendations  from  the  Aircraft  Certification  Procedures  issues  area  under 
the  Aviation  Rulemaking  Advisory  Committee  (ARAC).  The  recommendations 
included: 

•  Means  of  Compliance  with  Proposed  Quality  System  Requirements  (2-20-02); 

•  Production  Approval  Holder  Transition  to  New  Quality  System 
Requirements  (2-20-02); 

•  Quality  System  Guidance  (1 1-26-01); 

•  Proposed  Advisory  Circular  on  Standard  and  Commercial  Parts  (3-21-02); 

•  Organizational  Designated  Airworthiness  Representative  (3-21-02);  and 

•  Part  Design  Approval  Document  ( 1 1  -26-0 1 ) 

As  indicated  in  a  letter  to  you  earlier  this  year,  rulemaking  in  this  area  is  important  for 
several  reasons.  Most  important  are  enhancements  in  safety  and  system  efficiencies  for  1 1 
industry  and  Federal  Aviation  Administration.  Our  Rulemaking  Council  has  asked  the 
program  office  to  define  the  issues  that  should  be  in  this  rulemaking  effort  and  present  z 
plan  to  the  Council  for  approval  and  assignment  of  resources.  The  Council  will  address 
this  action  at  its  next  meeting. 

I  would  like  to  thank  the  aviation  community  for  its  commitment  to  ARAC  and,  in 
particular,  the  Production  Certification  and  Parts  Approval  Working  Groups  for  their 
expenditure  of  resources  to  develop  the  working  documents.  The  groups  are  commends 
for  their  extensive  deliberations  on  this  difficult  task.  /I  W 

li 


Signed  By 
Margaret  Gilligan 
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APPENDIX  D.  SELECTIONS  FROM  DOD  ACQUISITION  POLICY 


Item  1  -  Selected  portion  of  SECNAVINST  5000.2D 


SECNAVINST  5000. 2D 
October  16,  2008 


Chapter  7 

Systems  Engineering  and  Human  Systems  Integration 


References : 


(a)  POD  Directive  5000.1,  The  Defense  Acquisition 

System,  of  12  May  03 

(b)  DOD  Instruction  5000.2,  Operation  of  the  Defense 

Acquisition  System,  of  12  May  03 
(C)  OPNAVINST  3560 ,16A 

(d)  SECNAVINST  4-40.2 

(e)  DOD  4140 ,1-R,  DoD  Supply  Chain  Material 

Management  Regulation,  of  23  May  03 

(f )  ISO  9001  Quality  Management  Systems  - 
Requirements  Supplemented  by  AS 91 00 
International  Aerospace  Quality  Standard 


Chapter  7 

7.1 

7.1.1 

7.1.2 

7.1.3 

7.1.4 

7.1.5 

7.1.6 
7. 1.6.1 
7-1-6-2 


Systems  Engineering  and  Human  Systems  Integration 

Systems  Engineering 
Manufacturing  and  Production 
Quality 

Acquisition  Logistics  and  Sustainment 
Open  Architecture 

Reliability,  Availability,  and  Maintainability  (RAM) 
Interoperability  and  Integration 
IT  Design  Considerations 

DoD  Architecture  Franeworb/Defense  Information 


Item  2  -  Selected  portion  of  SECNAVINST  5000. 2D,  paragraph  7.1.2: 

7.1.2  Quality 

The  quality  program  should  ensure  the  use  of  best  engineering,  design, 
manufacturing  and  management  practices  that  emphasize  the  prevention  of  defects. 
Quality  should  be  designed  into  the  product  through  the  systems  engineering  design 
process  to  define  the  product  and  process  quality  requirements.  Contractors  should 
propose  a  quality  management  process  that  meets  required  program  support  capabilities. 
The  quality  management  system  may  be  based  on  the  fundamentals  described  in  the  ISO- 
9001  series  supplemented  by  AS9100,  International  Aerospace  Quality  Standard,  which 
provide  a  basic  minimum  quality  management  system  model. 

Additional  advanced  quality  requirements  should  be  considered  for  systems  based 
on  factors  such  as  risk,  design  complexity,  and  maturity,  process  complexity  and 
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maturity,  safety,  and  economics.  An  advanced  quality  management  system  builds  on  a 
basic  quality  management  system,  especially  during  the  design  /  development  phase,  by 
identifying  critical  product  and  process  characteristics,  design-to-manufacturing  process 
capabilities,  design  for  assembly  and  manufacturing,  design  to  control  process  variability, 
process  controls,  continuous  improvements,  etc.  The  quality  management  approach 
should  include  an  assessment  of  the  contractor’s  quality  management  process  and  its 
implementation,  including  those  related  to  assessments  or  oversight  of  subcontractors, 
suppliers,  and  special  process  facilities  (e.g.,  heat  treatment).  The  quality  management 
system  should  provide  timely  notification  and  feedback  to  contracting  and  program 
offices  in  areas  such  as  major  and  critical  deficiencies,  potential  manufacturing  process 
problems,  and  subcontractor,  supplier,  or  special  process  facilities  problems  that 
potentially  impact  the  program. 


Item  3  -  Selected  portion  of  SECNAVINST  5000.2E,  paragraph  6.1.2: 

6.1.2  Quality 

A  process  shall  be  in  place  to  assure  product  quality  during  design,  development, 
manufacturing,  production,  and  sustainment.  Quality  is  determined  by  the  extent  that 
products  and  services  meet  requirements  and  satisfy  the  customer  at  an  affordable  cost.  A 
quality  management  system  should  monitor,  measure,  analyze,  control  and  improve 
processes.  Quality  practices  and  quality  requirements  consistent  with  program  complexity 
and  criticality  shall  be  used  to  assist  in  reducing  risk,  assuring  quality,  and  controlling 
costs. 


Reference  (f)  is  a  model  for  quality  management  systems.  Contractors  may 
propose  alternative  systems,  as  long  as  they  are  found  technically  acceptable  by  the 
SYSCOM  technical  authority  and  accomplish  program  objectives. 


Item  4  -  Select  portions  of  the  Defense  Acquisition  System 

DODI  5000.02  policy  statements  related  to  manufacturing  development  with  the 

underlined  portions  added  for  emphasis:  (USD[AT&L]  2015) 

Selection  1 

(d)  EMD  Phase  Completion.  The  EMD  Phase  will  end  when:  (1)  the 
design  is  stable;  (2)  the  system  meets  validated  capability  requirements 
demonstrated  by  developmental  and  initial  operational  testing  as  required 
in  the  TEMP;  (3a)  manufacturing  processes  have  been  effectively 
demonstrated  and  are  under  control;  (3b)  software  sustainment  processes 
are  in  place  and  functioning;  (4)  industrial  production  capabilities  are 
reasonably  available;  and  (5)  the  system  has  met  or  exceeds  all  directed 
EMD  Phase  exit  criteria  and  MS  C  entrance  criteria.  EMD  will  often 
continue  past  the  initial  production  or  fielding  decision  until  all  EMD 
activities  have  been  completed  and  all  requirements  have  been  tested  and 
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verified  (26). 


Selection  2 

(10)  MS  C 

(a)  MS  C  and  the  Limited  Deployment  Decision  are  the  points  at  which  a 
program  or  increment  of  capability  is  reviewed  for  entrance  into  the  P&D 
Phase  or  for  Limited  Deployment.  Approval  depends  in  part  on  specific 
criteria  defined  at  Milestone  B  and  included  in  the  Milestone  B  ADM.  The 
following  general  criteria  will  normally  be  applied:  demonstration  that  the 
production/ deployment  design  is  stable  and  will  meet  stated  and  derived 

requirements  based  on  acceptable  performance  in  developmental  test 

events;  an  operational  assessment;  mature  software  capability  consistent 
with  the  software  development  schedule;  no  significant  manufacturing 
risks;  a  validated  Capability  Production  Document  (CPD)  or  equivalent 
requirements  document;  demonstrated  interoperability;  demonstrated 
operational  supportability;  costs  within  affordability  caps;  full  funding  in 
the  FYDP;  properly  phased  production  ramp  up;  and  deployment  support 
(27). 


Selection  3 

(12)  Full-Rate  Production  Decision  or  Full  Deployment  Decision.  The 
MDA  will  conduct  a  review  to  assess  the  results  of  initial  OT&E,  initial 
manufacturing,  and  limited  deployment,  and  determine  whether  or  not  to 

approve  proceeding  to  Full-Rate  Production  or  Full  Deployment. 
Continuing  into  Full-Rate  Production  or  Full  Deployment  requires 

demonstrated  control  of  the  manufacturing  process,  acceptable 
performance  and  reliability,  and  the  establishment  of  adequate  sustainment 
and  support  systems  (29). 


Selection  4 

Selection  4  comes  from  the  DODI  5000.02  and  the  SE  section  found  in  this 
document’s  enclosure  3,  paragraph  10  entitled  “Manufacturing  and  Producibility.”  The 
DODI5000.02  states: 

During  the  Engineering  and  Manufacturing  Development  Phase,  program 
managers  will  assess  the  maturity  of  critical  manufacturing  processes  to 
ensure  they  are  affordable  and  executable.  Prior  to  a  production  decision, 
the  Program  Manager  will  ensure  manufacturing  and  producibility  risks 

are  acceptable,  supplier  qualifications  are  completed,  and  any  applicable 
manufacturing  processes  are  or  will  be  under  statistical  process  control 
(84). 

Selection  5 
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Naval  Air  System  Command  Instruction  (NAVAIRINST)  in  NAVAIRINST  4355. 19D, 
(NAVAIR  2008)  (underlining  for  emphasis  added): 


The  Production  Readiness  Review  (PRR)  is  an  examination  of  a  program 
to  determine  if  the  design  is  ready  for  production  and  the  producer  has 
accomplished  adequate  production  planning  without  incurring 
unacceptable  risks  that  will  breach  thresholds  of  schedule,  performance, 
cost,  or  other  established  criteria.  The  full,  production-configured  system 
is  evaluated  to  determine  that  it  correctly  and  completely  implements  all 
system  requirements,  and  whether  the  traceability  of  final  system 
requirements  to  the  final  production  system  is  maintained.  At  this  review 
the  IPT  shall  also  review  the  readiness  of  the  manufacturing 

processes,  the  Quality  System,  and  the  production  planning,  i.e., 

facilities,  tooling  and  test  equipment  capacity,  personnel  development  and 
certification,  process  documentation,  inventory  management,  supplier 
management,  etc.  A  successful  review  is  predicated  on  the  IPT’s 
determination  that  the  system  requirements  are  fully  met  in  the  final 
production  configuration,  and  that  production  capability  form  a 
satisfactory  basis  for  proceeding  into  LRIP  and  FRP  (124). 


Selection  6 


Naval  Air  System  Command  Instruction  (NAVAIRINST)  in  NAVAIRINST  4355. 19E, 
(NAVAIR  2015) 

Criteria  PRR: 

Production  baseline,  Manufacturing,  Producibility  and  Quality 
requirements  are  producible  as  verified  by  the  results  of  the  Incremental 
Production  Readiness  Reviews  (iPRR) 

Rationale: 

Ensures  that  test  data  indicate  readiness  for  production;  ensures  the 
specified  manufacturing  and  quality  requirements  are  captured  in  the 
production  plans. 

Selection  7 

DODI  5000.02.  See  selection  6  and  Table  2: 


TECHNOLOGY  READINESS  ASSESSMENT  (TRA) 
SEC.  205,  P.L.  111-23  (Ref.  (an))  ASD(R&E) 
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STATUTORY  A  preliminary  assessment  is  due  for  the  Development  RFP 
Release  Decision  Point.  The  Assistant  Secretary  of  Defense  for  Research 
and  Engineering  (ASD(R&E))  will  conduct  an  independent  review  and 
assessment  of  the  TRA  conducted  by  the  Program  Manager  and  other 
factors  to  determine  whether  the  technology  in  the  program  has  been 
demonstrated  in  a  relevant  environment.  The  assessment  will  inform  the 
2366b  CERTIFICATION  MEMORANDUM  at  Milestone  B  (in 
accordance  with  10  U.S.C.  2366b  (Reference  (g)).  The  TRA  at  MS  C  is  a 
Regulatory  requirement  when  MS  C  is  Program  Initiation  (57). 
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APPENDIX  E.  DOD  TECHNICAL  WARRANTS 


The  following  is  a  white  paper  written  by  William  Ireland,  at  the  Naval 
Postgraduate  School  and  submitted  on  April  2,  2017,  in  support  of  his  thesis,  “Selection 
of  an  Alternative  Production  Part  Approval  Process  to  Improve  Weapon  Systems 
Production  Readiness.” 


Discussion 

In  the  DOD,  there  is  technical  authority  (TA)  that  interacts  with  programmatic 
authority  over  the  life  cycle  phased  weapons  systems  acquisition  process.  The  manner  of 
this  interaction  is  through  organizational  alignment  by  competencies.  Tomaiko,  in  his 
2008  thesis,  discusses  the  TA  framework  within  the  NAVESEA  SYSCOM  and  the 
development  of  a  new  specialized  certification.  The  SYSCOM  TA  comes  from  a 
competency-aligned  delegation  of  authority.  The  effectiveness  of  introducing  a  new  TA 
is  explored: 

In  2006,  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition,  ASN  (RD&A),  mandated  the  transformation  of  Naval  Sea  Systems 
Command  (NAVSEA)  into  a  Competency-Aligned  Organization  (CAO).  A  CAO  fosters 
a  competency-based  approach  to  mission  performance.  A  key  objective  of  NAVSEA’s 
new  CAO  is  to  improve  program  management  authority  and  contract  authority  through 
more  effective  technical  authority.  A  key  challenge  facing  NAVSEA  in  establishing  a 
new  CAO  is  aligning  program  management,  contract,  and  technical  competencies.  This 
will  require  a  common  alignment  of  the  engineering  workforce  across  the  Navy,  as  well 
as  common  policy  development  and  implementation.1 

The  TA  approach  is  the  subject  of  SECNAVINST  5400. 15C  dated  September  13, 
2007  and  as  updated  and  changed  December  2,  2011. 2  The  organizational  construct 
representing  TA  is  defined  along  technical  workforce  competencies.  These  organizational 
constructs  are  delegated  authorities.  Each  competency  is  recognized  as  an  authority  that 
flows  down  through  its  supervisory  chain.  These  chains  of  authority  are  further  described 
in  various  policy  documents  and  practices  within  the  respective  SYSCOMs.  The 
Secretary  of  the  Navy  (SECNAV)  delegates  the  TA  through  the  operational  Navy 
(OPNAV)  and  the  SYSCOMs  to  lower  level  experts  identified  as  Technical  Warrant 
Holders  (TWH)  for  certain  key  performance  areas.  The  NAVAIR  SYSCOM  uses  a  CAO 
that  follows  their  technical  workforce  competencies  with  few  actual  TWHs  while 
NAVSEA  SYSCOM  follows  a  more  commodity  based  designation  with  many  TWHs. 

Core  beliefs  in  the  technical  authorities  construct,  as  discussed  by  Tomaiko,  is 
based  on  an  assumption  that  asserts  that  a  well  behaved  process  will  result  in  quality 

1  IMPROVING  THE  U.S.  NAVY’S  EXECUTION  OF  TECHNICAL  AUTHORITY  THROUGH  A  COMMON  RISK 
MANAGEMENT  AND  TECHNICAL  ASSESSMENT  PROCESS  by  Thomas  Andrew  Tomaiko,  2008. 

2  SECNAVINST  5400. 15C  CH-,  1  ASN  (RD&A),  September  13,  2007  and  2  December  2011 
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acquisition  outcomes.  Enablers  of  the  CAO  TA  approach  is  found  in  required  periodic 
training  and  commitment  levied  on  key  acquisition  personnel  in  order  to  maintain 
continuity  of  program  oversight  and  execution. 

Practices  within  certain  valued  threads  of  authority,  such  as  safety,  requires  an 
independent  communication  path  to  leadership  and  milestone  decision  authorities. 
Typically,  there  is  a  reporting  of  status  of  significant  technical  issues  to  the  SYSCOM 
commander.  One  example  of  a  critical  assessment  related  to  safety  examines  flight 
worthiness.  The  flight  worthiness  assessment  is  a  requirement  that  informs  leadership  by 
issuing  a  certificate  of  conformance  or  its  denial.  In  this  case,  the  NAVAIR  SYSCOM 
conducts  gate  reviews  over  an  item’s  life  cycle  acquisition  development  and  assures  the 
proper  certificates  are  completed  prior  to  Initial  Operational  Capability  (IOC). 

Certifications  used  to  communicate  compliance  to  key  performance  capabilities  is 
based  upon  the  item  being  able  to  show  compliance  objectively  to  accepted  standards  and 
reporting  policies.  SECNAVINST  5400.15  C  states  the  certificate  responsibility  as 
coming  from  the  CNO  /  CMC  and  delegated  to  the  SYSCOMs,  PEOs  and  DRPMs. 

Technical,  Programmatic  and  Certification  Authority 

Authorities,  technical  or  programmatic,  are  described  in  the  key  policy  documents 
defining  the  requirements  for  each  SYSCOM  derived  from  the  SECNAVINST  5400-C, 
dated  2  DEC  2011.  The  instruction  defines  TA  with  its  roles  and  responsibilities.  The  use 
of  the  terms  that  describe  the  role  of  a  TA  and  certification  follows  closely  a  dictionary 
definition  giving  the  common  meaning  of  warrant.  The  actual  exercise  of  granting  a 
warrant  and  communicating  compliance  requires  standards  that  show  required  adherence 
evidence  in  order  to  adjudicate  a  specific  certification’s  requirements.  Those  involved  in 
the  technical  and  programmatic  authority  chain  are  responsible  for  issuing  and 
communicating  any  warrant  to  decision  makers.  Refer  to  the  definitions  of  authorities  in 
DOD  acquisition  and  the  general  meaning  of  warrant: 

Definition  1: 

(3)  Technical  authority  (TA).  TA  is  the  authority,  responsibility,  and 
accountability  to  establish,  monitor  and  approve  technical  standards,  tools,  and 
processes  in  conformance  with  applicable  Department  of  Defense  (DOD)  and 
DON  policy,  requirements,  architectures,  and  standards. 

Definition  2: 

Technical  authority  is  the  authority,  responsibility  and  accountability  to  establish, 
monitor  and  approve  technical  products  and  processes  are  in  conformance  to 
higher  authority  policy,  requirements,  architectures  and  standards. 

Programmatic  authorities  manage  all  aspects  of  assigned  programs  from  concept 
to  disposal,  including  oversight  of  cost,  schedule  and  performance,  and  direction 
of  life  cycle  management. 

Certification  authority  is  a  special  case  of  technical  authority  where  there  is 
authority  to  certify  that  products  meet  established  standards.  Specific  certification 
authority  is  defined  by  the  technical  process  documentation  established  by  the 
cognizant  technical  authority. 
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Definition  3: 

Select  portions  for  the  definition  of  “warrant”3 
war- rant  (wor'ont,  wdr'-)  n. 

Something  that  provides  assurance  or  confirmation;  a  guarantee  or  proof 

Authorization  or  certification;  sanction,  as  given  by  a  superior. 

a.  A  warrant  officer. 

b.  A  certificate  of  appointment  given  to  a  warrant  officer. 

To  capture  briefly  the  dual  role  of  a  TWH  there  is  the  definition  of  a  technical 
authority  as  associated  to  a  workforce  competency  or  office  and  the  reporting  of 
compliance  to  technical  standards  for  key  performance  areas  with  certification  or 
warrants: 

Technical  Warrant  Holder:  Significant  delegated  chain  of  authorities  to  a  lead 
office/officer. 

Certification  or  Warrant  Process:  Actions  that  lead  to  documenting  compliance  to 
the  related  authority  and  resultant  certification  or  warrant  granted  within  the 
domain  of  the  authority. 

Types  of  Technical  Authority  as  Warrant  Holders 

With  respect  to  types  of  authorities  in  this  discussion  is  the  TA  that  is  delegated. 
Consider  the  SECNAVINST  5400. 15C: 

The  SYSCOM  Commanders  are  responsible  for:  providing  for  in-service  support; 
providing  support  services  to  PEOs  and  DRPMs  without  duplicating  their  management 
functions;  and  serving  as  the  technical  authority  and  operational  safety  and  assurance 
certification  authorities  for  their  assigned  areas  of  responsibility. 

As  a  practical  matter,  this  delegation  of  TA  is  conferred  to  certain  Chief  Systems 
Engineers  (CSEs)  or  the  SYSCOM ’s  Chief  Engineer  (CHENG)  by  the  commanders  of 
the  SYSCOMS  to: 

“(5)  Exercise  Technical  Authority  and  certification  authority  for  weapon  and  IT 
systems.  ” 

One  example  of  delegated  TA  is  from  the  NAVAIR  SYSCOM,  where  the 
CHENG  for  aviation  manages  the  aviation  baseline  for  Ship  Design  Managers  (SDMs). 
In  another  case,  the  Cost  Engineering  Managers  (CEMs)  ensure  independent  cost 


3  The  Free  Dictionary:  URL:  http://www.thefreedictionary.com/warrant,  9/29/2016. 
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engineering  and  estimating  in  support  of  Navy  programs.  Lastly,  there  is  the  Technical 
Area  Expert  (TAE)  acting  under  the  top  CHENG  as  the  Navy’s  expert  in  the  assigned 
technical  domain  (e.g.  NAVAIR  -  Air  Vehicles;  NAVSEA  -  Shipboard  Pumps).  Further, 
there  have  been  Technical  Process  Owners  (TPOs)  who  provide  definition  and 
documentation  for  the  assigned  technical  processes  (e.g.  NAVAIR  -  Air  Worthiness 
Certification).  NAVSEA  will  define  Waterfront  and  Depot  Chief  Engineers  (CHENGs) 
and  lead  the  technical  efforts  of  the  SYSCOMs  from  the  waterfront  and  depots. 

Technical  Authority  Policy  and  Manufacturing  Competency 

TA,  as  it  applies  to  NAVAIR,  NAVSEA,  SPAWAR,  NAVFAC,  and  NAVSUP,  is 
derived  by  DOD  TA  policy  that  defines  various  types  of  Technical  Warrant  Holders 
(TWH).  There  are  specific  policies  and  instructions  that  define  the  roles  for  these 
technical,  programmatic  and  certification  authorities  within  the  SYSCOMs.  First,  a  TA  is 
conferred  upon  through  a  competency  office  holder  or  individual  who  is  considered  a 
Technical  Warrant  Holder  (TWH). 

The  TWH  is  chartered  with  establishing  specific  policy  for  resolving  conflicts  on 
technical  decisions  within  their  domain  of  delegated  expertise  and  oversight 
responsibility.  In  addition,  these  authorities  are  assigned  to  support  DOD  acquisition  to 
assure  products  meet  requirements  and  obtain  any  necessary  certifications  and  approvals 
to  advance  a  product’s  development  through  the  various  gated  technical  reviews  over  the 
DOD  life  cycle  acquisition  phases.  As  stated  in  the  OPNAV  Instruction  (OPNAVINST 
5450.350),  this  TA  is  deployed  to  the  NAVAIRSYSCOM  stating  the  TWH 
responsibility:4 

(5)  Exercise  Technical  Authority  and  life  cycle  management  for  assigned 
programs  and  oversee  core  processes,  operational  safety,  and  assurance 
certification  required  to  support  the  acquisition,  in-serx’ice  support,  and  disposal 
of  weapon  and  IT  systems. 

In  addition,  the  OPNAV  INST  confirms  the  relation  to  a  competency  as  long  as  it 
is  not  chartered  to  another  organization  in  the  requirement  stating: 

(b)  Management  of  shore  activities,  industrial  management  of  depot  maintenance 
activities,  administration  of  DOD  policies  on  manufacturing  methods  and 
technology  and  metrication; 

Technical  Authority  for  Manufacturing  in  DOD  Acquisition 

The  SYSCOMS  are  similar  to  one-another  in  their  application  of  TA  within  their 
respective  CAO  structures.  The  NAVAIR  TA  for  manufacturing  is  derived  from  the 
systems  engineering  competency.  The  NAVAIR  CHENG  further  delegates  the  TA  for 
manufacturing  to  the  division  head  of  the  Manufacturing  and  Quality  organization  as  the 
workforce  expertise  for  manufacturing.  In  this  chain  of  authority,  representing 
specifically  the  TA  for  manufacturing,  the  instruction  does  not  identify  a  requirement  for 
certification  to  warrant  a  contractor’s  readiness  to  produce.  When  a  warrant  or 
certification  of  compliance  is  defined  within  the  TA  there  are  associated  standards  and 


4  OPNAVINST  5450.350  dated  24  Apr  2012 
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requirements  that  must  be  satisfied  representing  a  key  performance  area.  However,  for 
Manufacturing  and  Quality  it  has  fallen  short  of  being  a  key  performance  area  requiring 
certification.  There  are  no  standards  related  to  manufacturing  or  a  production  readiness 
decision  requiring  a  certification  for  production  approval  in  the  DOD  acquisition.  This 
lack  of  a  standard  manner  to  demonstrate  process  capability  was  observed  by  the  GAO 
report  on  “Select  Weapon  Systems  reporting  that  there  was  a  lack  of  a  standard  approach 
for  contractor’s  compliance  for  production  readiness.  A  standard  would  be  needed  for 
production  readiness  if  process  capability  would  become  a  key  performance  area  for  TA 
certification. 

Current  approval  for  production,  in  the  NAVSEA  case,  acquisition  practices 
follow  a  technical  review  process  providing  technical  ownership  across  commodities  of 
interest  for  its  production  readiness  oversight.  Presently,  in  NAVSEA,  there  are  about 
250  different  specific  TWHs  representing  particular  commodities.  In  the  NAVAIR  case, 
the  acquisition  review  process  relies  upon  an  approach  that  assesses  an  end  item’s 
Production  Readiness  Review  (PRR).  The  technical  review  process  at  PRR  uses 
relatively  few  TWHs  associated  with  workforce  competencies,  relying  primarily  on  the 
TWH  for  manufacturing  as  to  reviewing  and  reporting  of  risk  with  respect  to  production 
readiness. 

In  either  case,  the  technical  assessments  of  production  readiness  occur  prior  to 
Milestone  C’s  production  decision.  At  MS  C,  a  programmatic  authority  will  then  report  if 
there  are  any  significant  manufacturing  risks  to  decision  authorities  in  a  program’s 
capabilities  production  document  (CPD)  given  to  the  MDA.  While  there  is  a  Production 
Readiness  Review  used  to  report  readiness  risk,  there  are  no  policies,  instructions  or 
standards  of  compliance  within  the  DOD  manufacturing  competency  that  could  be  used 
to  certify,  with  a  warrant,  such  readiness. 

It  is  important  to  note  that  the  Secretary  of  Defense  (SECDEF),  through  the  chain 
of  Technical  Authorities,  identified  “Manufacturing,”  Integrated  Logistics  Support  and 
T&E  as  equals  in  its  instruction  and  guidance  but  only  the  manufacturing  domain  does 
not  require  a  warrant  as  evidence  demonstrating  a  compliance  for  readiness.  The 
manufacturing  performance  area  for  oversight,  by  a  TWH,  was  taken  from  the 
SECNAVINST  5400. 1 5C  dated  September  2007 : 

(1)  Oversee  the  core  processes  required  to  support  the  acquisition,  in-service 
support,  and  disposed  of  weapon  and  IT  systems.  Core  processes  include:  (a) 
Realistic  and  reasonable  cost  estimating;  (b)  Technology  development  and 
technical  readiness  assessment;  (c)  Systems  engineering  (acquisition  and  in- 
service)  and  development,  including  Environment,  Safety  and  Occupational 
Health  (ESOH)  management;  (d)  Manufacturing;  (e)  Test  and  evaluation;  (f)  ILS 
(acquisition  and  in-service);  (g)  Installation;  (h)  Maintenance  and  modernization 
planning;  (i)  Configuration  management; 

In  the  current  instruction,  2  December  2011  release,  the  manufacturing  thread  is 
somewhat  ambiguous  to  TA  delegation  and  oversight  responsibility.  On  the  other  hand, 
technical  readiness  of  an  item  relies  upon  qualification  testing  that  demonstrates  key 
functional  performance  and  holds  a  much  higher  standard  of  review  that  is  observed.  One 
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example  is  the  Technological  Readiness  Level  (TRL)  required  by  public  law  requiring  a 
TRL  level  6  prior  to  EMD.5  Additional  functional  performance  verification  prior  to 
production  often  requires  a  demonstration  of  functional  requirements  under  qualification 
testing  and  Operational  Test  and  Evaluation  (OT&E).  Representative  functional 
performance  requirements  and  certifications  are  required  to  be  documented  in  order  to 
satisfy  a  production  decision  leading  to  Initial  Operation  Capability  (IOC)  and  will 
include  special  certifications  related  to  that  end  item. 

Requirements,  in  the  MRL  and  PRR  sense,  are  not  under  certification  reporting 
and  there  is  no  warrant  to  report  readiness  compliance  demonstrating  a  production 
system’s  process  capability.  One  aspect  of  the  2  December  2011  change  to  the 
SECNAVINST  5400.15  that  shows  the  technical  manufacturing  thread  in  the  TA 
obligation  removed: 

(1)  Exercise  management  authority,  including  selection  and  application,  over 
core  capabilities  that  support  the  SECNAVINST  5400.1 5C  acquisition,  in-service 
support,  and  disposal  of  assigned  weapons  and  IT  systems.  These  capabilities 
include:  (a)  Business  and  financial  management;  (b)  Life  cycle  logistics;  (c)  Test, 
evaluation  and  certification;  (d)  Technology  evaluation(s);  (e)  Systems 
engineering  ( including  ESOH  management);  (f)  Installation,  maintenance,  and 
modernization;  (g)  Configuration  management;  and  (h)  Demilitarization  and 
disposal. 

When  contrasting  the  DOD  production  readiness  practices,  to  the  automotive 
industry,  one  finds  the  automobile  industry  uses  a  model  of  compliance  requiring  a 
manufacturing  readiness  /  production  readiness  defined  by  a  certificated  warrant  process 
for  production  approval.  Other  industries,  such  as  FDA  and  the  FAA,  all  require  a 
producer  to  demonstrate  production  readiness  knowledge  prior  to  production 
commitment  by  issuing  a  warrant  as  part  of  a  Production  part  approval  process  (PPAP). 
The  notion  of  issuing  a  certification  that  warrants  a  demonstrated  capable  production 
process  is  not  a  structure  in  the  DOD  CAO.  As  a  model,  the  PPAP  is  a  long  standing  best 
practice  in  the  automotive  industry  and  serves  as  a  key  performance  area  standard.  The 
GAO  013  report  was  in  support  of  creating  a  production  readiness  standard  for  DOD 
production  approval,  stating  there  is  a  need  to  look  at  process  capability  demonstration  in 
a  standardized  manner.6  In  2010,  this  process  of  certification  for  production  readiness 
was  also  recommended  for  use  in  the  DOD  by  the  National  Defense  Industrial 
Association  (NDIA)  (NDIA  2010,  pl9).7  Consider  what  is  said  by  Tomaiko  with  regards 
to  the  need  to  develop  and  standardize  the  TA  process  on  interoperability  in  DOD 
acquisition  -  Quoted  directly  (Tomaiko,  2008): 


6  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 

6  Defense  Acquisitions  -  Assessments  of  Selected  Weapon  Programs.  RTC  (Report  No.  GAO-13-294SP)  Washington, 
DC:  General  Accountability  Office,  2013. 

7 

NDIA  Gulf  Coast  Chapter,  2010.  Acquisition  Excellence  through  Effective  Systems  Engineering;  Systems  Engineering 
Deficiencies  and  Corrections  for  Air-Launched  Tactical  Weapons.  Panel  Report,  Arlington,  VA:  NDIA. 
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This  thesis  achieved  its  purpose  of  improving  execution  of  Technical  Authority  by 
defining  the  relationship  between  program  authority  and  Technical  Authority  and 
describing  how  to  assess  and  improve  the  state  of  Technical  Authority  through 
common  policy  development  and  implementation.  Still,  more  work  needs  to  be 
done.  Future  research  necessary  to  help  the  SYSCOMs  implement  a  common  risk 
management  process  includes  development  and  deployment  of  an  Integrated 
Assessment  Tool  (IAT).  Future  research  cdso  needs  to  include  promulgating  a 
common  policy  for  developing  Systems  Engineering  Plans,  a  common  technical 
review  process,  a  common  total  platform  and  interoperability  certification 
process,  and  a  common  systems  engineering  training  program. 

In  the  case  brought  by  Tomaiko,  a  more  effective  TA  process  was  desired  for  the 
key  performance  area  of  interoperability.  Fortunately,  the  TA  on  interoperability 
followed  the  recommendation  discussed  by  Tomaiko.  Now  there  is  a  complete 
certification  process  improvement  in  practice.  Compliance  for  the  certificated 
Interoperability  requirements  are  given  in  the  Interoperability  Process  Guide,  dated  10 
September  2012.  The  guide  is  managed  by  the  Defense  Information  Systems  Agency 
(DISA).8 

Rationale  for  a  DOD  PPAP  Warrant  /  Certificate 

The  automotive  industry  production  approval  process  observes  a  stable 
commercial  best  practice  giving  acquisition  oversight  assuring  a  production  part  is  ready 
to  start  production  based  on  a  set  of  PPAP  standards.  In  the  commercial  case,  a  PPAP 
warrant  holder  represents  that  a  production  system  of  interest  produces  parts  meeting 
requirements.  This  process  is  applied  throughout  an  entire  supply  network  in  the  exact 
same  manner.  The  PPAP  can  be  assembled  from  a  bottom-up  approach  providing 
consistent  evidence  of  compliance. 

The  automotive  industry  has  relied  upon  the  PPAP  as  a  performance  of  a 
disciplined  systems  engineering  application  to  the  manufacturing  development  process 
and  production  ready  demonstration.  The  PPAP  warrant  process  is  flowed  from  the 
original  equipment  manufacturer  down  to  the  lowest  level  supplier  prior  to  production 
approval.  A  review  of  one  reference  in  DOD  guidance  issued  by  the  NAVAIR  SYSCOM 
in  the  NAVAIRINST  4355. 19D  one  finds  that  the  instruction  considered  a  very  limited 
view  of  certification  associated  only  to  personnel  in  support  of  production  readiness.9 
There  was  attention  to  functional  performance  demonstration  and  production  process 
readiness  and  planning  (highlighted  selections  by  the  author).  Production  capability  was 
to  be  assured  at  PRR.  The  requirements  do  not  develop  any  description  of  what  is  meant 
by  a  “satisfactory  basis  for  determining  readiness.”  If  it  was  to  be  from  objective 
manufacturing  requirements  demonstrated,  no  such  production  capability  requirements 
are  shown  to  be  required.  Reference  the  quote  from  NAVAIRINST: 


8  Interoperability  Process  Guide,  Version  1.0,  dated  10  September  2012,  Defense  Information  Systems  Agency 

9  NAVAIRINST  4355,  19E  FEB  06  2015. 
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The  Production  Readiness  Review  (PRR)  is  an  examination  of  a  program  to 
determine  if  the  design  is  ready  for  production  and  the  producer  has 
accomplished  adequate  production  planning  without  incurring  unacceptable  risks 
that  will  breach  thresholds  of  schedule,  performance,  cost,  or  other  established 
criteria.  The  full,  production-configured  system  is  evaluated  to  determine  that  it 
correctly  and  completely  implements  all  system  requirements,  and  whether  the 
traceability  of  final  system  requirements  to  the  fined  production  system  is 
maintained.  At  this  review  the  IPT  shall  also  review  the  readiness  of  the 
manufacturing  processes,  the  Quality  System,  and  the  production  planning ,  i.  e. , 
facilities,  tooling  and  test  equipment  capacity,  personnel  development  and 
certification,  process  documentation,  inventory  management,  supplier 
management,  etc.  A  successful  review  is  predicated  on  the  IPT’s  determination 
that  the  system  requirements  are  fully  met  in  the  fined  production  configuration, 
and  that  production  capability  form  a  satisfactory  basis  for  proceeding  into 
LRIP  and  FRP.  p.  124. 

In  the  current  NAVAIRINST  4355. 19E,  the  PRR  requirements  related  to 
production  and  manufacturing  readiness  do  not  include  a  “basis  of  satisfaction”  to  be 
determined  towards  a  production  capability.  The  writers  of  version  D  anticipated  some 
“basis  of  satisfaction”  to  be  determined  or  assured  but  did  not  go  as  far  as  requiring  an 
objective  demonstration  or  certification  as  recommended  by  the  NDIA  in  2010. 
Reference  the  table  on  PRR  requirements  from  NAVAIRINST  4355.19E:10 


Criteria  PRR 

Rationale 

c.  Production  baseline.  Manufacturing,  Producibility 
and  Quality  requirements  are  producible  as  verified 
by  the  results  of  the  Incremental  Production 
Readiness  Reviews  (iPRR) 

Ensures  that  test  data  indicate  readiness  for 
production;  ensures  the  specified  manufacturing  and 
quality  requirements  are  captured  in  the  production 
plans. 

In  the  DOD  technical  review  process,  there  is  a  system  engineering  review 
establishing  a  stable  and  manufacturable  design  at  the  Critical  Design  Review  (CDR). 
The  subsequent  PRR  review  is  the  technical  review  leading  to  production  start  at  MS  C. 
The  CDR  and  PRR  reviews  are  conducted  to  ensure  that  functional  test  data  and 
manufacturing  risk  indicate  readiness.  The  specified  manufacturing  and  quality  readiness, 
that  span  CDR  and  PRR,  rely  solely  upon  manufacturing  and  quality  planning  and  not  on 
production  capability  that  had  a  basis  of  being  satisfied  in  the  earlier  version  of  the 
instruction.  With  a  focus  on  the  manufacturing  review  requirements,  the  CDR  and  PRR 
are  simply  defined: 

CDR: 

System  and  subsystem  level  analyses  of  producibility,  manufacturing  process,  and 
process  controls,  [in]  support  [of]  the  product  baseline 


10  NAVAIRINST  4355. 19E,  19E  FEB  06  2015  p  42. 
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Ensures  that  contractors  have  processes  and  process  controls  in  place  to 
manufacture  systems  per  the  product  baseline 

PRR: 

Product  baseline,  manufacturing,  producibility  and  quality  requirements  are 
producible  as  verified  by  the  results  of  the  incremental  production  readiness 
reviews  (IPRR-  without  a  special  definition ) 

Summary  of  Technical  Authority  Discussion  in  Manufacturing 

The  manufacturing  solution  that  historically  led  the  automotive  industry  to  devise 
one  standard  approach  for  a  PPAP  warrant  was  the  industry’s  response  to  poor 
automotive  production  outcomes  in  the  80’ s  and  foreign  competition.  This  is  similar  to 
the  GAO’s  findings  that  reported  poor  production  outcomes,  in  the  DOD,  due  to  a  lack  of 
standard  and  demonstrated  process  capability  at  MS  C.  The  need  for  a  common  approach 
for  the  automotive  OEMs  limited  a  proliferation  of  OEM  standards  that  were  a  burden  on 
the  supply  network.  A  common  approach  emerged  that  employed  a  set  of  prescriptive 
quality  guidelines  and  a  production  approval  requirement  standard.  These  prescriptive 
standards  were  deployed  with  four  overlapping  activities  called  advanced  product  quality 
planning  (APQP).11 

The  APQP  is  used  to  guide  best  practices  in  the  development  of  a  manufacturing 
process.  If  this  approach  would  be  applied,  within  the  DOD  acquisition  and  TA 
constructs,  it  would  serve  to  significantly  reduce  time  and  costs  that  are  hard  to  assess 
with  the  endless  different  readiness  approaches  taken  by  contractors  and  their  suppliers. 
One  common  approach  would  streamline  and  enhance  production  oversight  and  be  able 
to  use  the  PPAP  standard  for  a  certification  warrant  to  realize  quality  improvement.  The 
adaptation  of  the  process  capability  using  APQP  and  PPAP  are  described  in  brief: 


APQP 

Advanced  Manufacturing  -  Prototype  (where  critical  characteristics  were 
identified )  &  Production  Planning  through  final  equipment  validation  (facilities 
planning  &  execution  /  equipment  &  process  including  sub-tier  entity  oversight  - 
all  PPAP)  early  process  prove-out  -  (relied  upon  process  potential ). 

PPAP 

Transitions  from  the  development  phase  to  the  production  phase  by  a  warranted 
production  readiness  demonstration.  Customer  acceptance  (relied  upon  process 
capability  -  where  critical  characteristics  were  controlled ). 

The  PPAP  serves  as  the  evidence  of  readiness  to  produce  and  is  conducted  prior 
to  production.  The  PPAP  is  a  handshake  from  advanced  manufacturing  to  plant 
operations.  A  simplification  of  the  PPAP  requirement:  a  production  part  that  meets 
functional  requirements,  has  been  manufacturing  from  a  defined  production  environment, 


* '  AIAG.  2008.  Advanced  Product  Quality  Planning  and  Control  Plan  (APQP)  2nd  ed.  Reference  Manual,  Advanced 
Product  Quality  Planning  Working  Group,  Southfield:  AIAG. 
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is  produced  under  manufacturing  controls,  is  run  at  production  rates,  parts  show  statistical 
a  demonstrated  capability  and  thereby  assure  production  readiness  at  production  start. 

Benefits  of  the  PPAP  Warrant  Process  Surveyed 

Surveys  in  the  late  1990s  by  the  American  Society  of  Quality  and  McGraw-Hill 
allowed  a  comparison  of  the  prescriptive  quality  systems  known  as  the  QS  9000  quality 
standards  and  the  ISO  9000  quality  standards  (including  AS9100)  (ISO  9000  was 
common  to  DOD  acquisition).12  The  prescriptive  standards  of  the  automotive  industry 
revealed  compelling  evidence  that  the  knowledge-based  approach  demonstrating  process 
capability  was  superior  to  non-prescriptive  quality  standards  based  on  the  ISO  9000 
series  alone.  The  automotive  prescriptive  quality  system  achieved  at  least  50% 
improvement  50%  of  the  time  based  on  reduced  scrap,  improved  reliability  and  improved 
profits.  The  ISO  9000  family  of  quality  standards  showed  that  only  6%  of  respondents 
improved  quality  by  adherence  to  the  standards.1314 

Aerospace  Standard  -  AS  9145 

The  success  of  the  automotive  industry  best  practices,  of  APQP  and  PPAP,  had 
not  gone  unnoticed  and  is  the  reason  why  there  is  a  new  aerospace  standard,  released 
11/2016,  to  adopt  the  APQP  and  the  PPAP,  reference  SAE  -  AS9145.15  The  APQP  and 
PPAP  are  processes  that  show  a  solution  space  that  has  demonstrated  success  for  over  25 
years  as  a  best  practice  in  the  automotive  industry  and  has  been  largely  adopted  by  the 
FAA  and  the  FDA.  Therefore,  there  is  a  need  to  develop  a  certification  process  for 
process  capability,  as  a  key  performance  area,  in  the  DOD  using  the  CAO  and  the  TA 
construct  to  close  the  manufacturing  knowledge  gap  reported  by  the  GAO.  This  would 
close  the  knowledge  gap  in  manufacturing  tied  to  poor  production  outcomes  from  given 
the  current  lack  of  standard  disciplined  demonstration  of  process  capability  prior  to 
production  start. 

Conclusion 

The  success  of  the  automotive  approach  relies  upon  a  third  party  consortium  - 
Automotive  Industry  Action  Group  (AIAG).  The  AIAG  has  trained  and  maintained  the 
knowledge  base  over  the  entire  supply  network  on  behalf  of  OEMs  since  the  early  1990s. 
This  success  of  APQP  and  PPAP  has  been  given  stature  for  consideration  as  a 
prescriptive  standard  for  the  aerospace  industry  by  the  Society  of  Automotive  Engineers. 
The  significantly  more  prescriptive  standard  AS9145  with  certification  warrant  is  now 
proposed  by  the  Society  of  Automotive  Engineers  on  behalf  of  the  aerospace  industry 


19. 

Naveh,  Eitan,  Alfred  Marcus,  and  Hyoung  Koo  Moon.  1999.  ISO  9000  Cost  -  Benefit  Survey.  Quality  management 
systems  Update  and  Plexus  Corporation,  New  York:  McGraw-Hill  Inc. 

1 3  AIAG/ASQ.  Quality  Survey  Results.  Survey  Workshop,  Novi:  Automotive  Industry  Action  Group/ American 
Society  of  Quality,  1997. 

14  AIAG.  1998.  1998  Annual  Quality  Survey  Report.  Survey,  Southfield:  Automotive  Industry  Action  Group. 

13  Society  of  Automotive  Engineers.  2015.  Advanced  Product  Quality  Planning  (APQP)  /  Production  Part  Approval 
Process  (PPAP).  Issuing:  G-14  Americas  Aerospace  Quality  Standards  Committee  SAE  International.  Host:  SAE.org 
URL:  http//standards. sae.org/as9145  (accessed  April  3,  2017). 
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because  that  standards  body  recognized  the  superior  production  outcomes  over  the  use  of 
the  ISO  and  AS9100  quality  systems  standards. 

Therefore,  this  paper  urges  the  adaptation  of  APQP  and  PPAP  for  DOD 
acquisition  process  improvement  identifying  Production  Readiness  as  a  key  performance 
area  requiring  a  certification  warrant  at  MS  C.  These  standards  and  guidelines  for  quality 
and  manufacturing  readiness  are  able  to  be  applied  within  the  framework  of  the  Code  of 
Federal  Regulations  but  would  need  the  DOD  to  update  their  policy  and  guidance  with 
respect  TA  roles  and  responsibilities  for  manufacturing. 
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APPENDIX  F.  DEALING  WITH  MULTIPLE  QMS 


When  there  is  more  than  one  quality  management  system,  to  satisfy  multiple 
customer  requirements,  a  supplier  may  adopt  an  approach  that  addresses  each.  This  can 
occur  when  they  serve  customers  that  cross  industries  such  as  FAA,  FDA  and  automotive 
that  has  industry  specific  standards  for  their  quality  management  approach.  If  a  supplier 
serves  a  DOD  contractor,  then  there  are  unique  requirements  that,  in  general,  follow 
either  ISO  9000  or  AS  9000  requirements  for  Quality  Management  Systems.  One  quality 
system  may  try  to  satisfy  all  such  standards  by  a  consolidation  approach  first  addressing 
the  common  requirements  of  ISO  9000  and  then  with  additional  requirements  treated  in 
some  unique  manner  such  as: 


The  Company  Quality  Management  System  shall  meet  the  requirements  of  the 
International  Standard  ISO-9001 :2008. 

Additional,  for  products  sold  to  automotive  application,  the  company  quality 
management  system  shall  comply  with  the  requirements  of  the  ISO/TS  16949  standard,  as 
it  appears  in  italic  type  with  a  (TS)  preceding  the  statement. 

Additional,  for  products  sold  as  an  aerospace  application,  the  company  quality 
management  system  shall  comply  with  the  requirements  of  the  AS-9100  standard,  as  it 
appears  in  italic  type  with  an  (AS)  preceding  the  statement. 


Note:  How  to  address 
personal  knowledge. 


multiple  quality  management  systems  given  by  the  author’s 
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APPENDIX  G.  DOD  MEMO  TO  GAO 


This  memo  from  the  DOD  to  the  GAO  is  an  example  response  to  the  findings  of  a 
GAO  report  on  weapon  programs. 


DoD  Response  to  GAO-10-439  Recommendations 

GAO  DRAFT  REPORT  DATED  MARCH  12,  2010 
GAO-10-439  (GAO  CODE  120793) 

BEST  PRACTICE:  DOD  CAN  ACHIEVE  BETTER  OUTCOMES  BY 
STANDARDIZING  THE  WAY  MAYUFAC  TURING  RISKS  ARE  MANAGED" 

DEPARTMENT  OF  DEFENSE  C  OMMENTS  TO  THE  GAO 
REC  OMMENDATIONS 

RECOMMENDATION  1:  Tie  GAO  recommends  that  the  Secretary  of  Defense  reqmre 
the  assessment  of  manufacturing  readiness  across  DoD  programs  using  consistent  MRL 
criteria  as  a  basis  for  measuring,  assessing,  reporting,  and  c  omnium  eating  manufacturing 
readiness  and  risk  on  science  and  technology  transition  projects  and  acquisition 
programs. 

DOD  RESPONSE:  Partially  concur:  The  Department  of  Defense  recognizes  that  mature 
manufacturing  processes  and  readiness  are  critical  to  achieving  predictable  and  successful 
program  outcomes  It  also  recognizes  the  value  m  assessing  manufacturing  nsks  during 
science  and  technolog)'  research  on  technologies  planned  to  be  incorporated  into 
acquisition  programs.  Department  of  Defense  Instruction  (DoDI)  5000.02,  Operation  of 
the  Defense  Acquisition  System,  dated  S  December  2008  reflects  an  increased  focus  on 
manufacturing  throughout  the  acquisition  lifecycle  for  programs  of  all  acquisition 
categories.  Specifically,  it  establishes  a  framework  to  continually  assess  and  mitigate 
manufacturing  risks  during  the  Analysis  of  Alternatives.  2365k  certifications  to  Congress. 
Preliminary  and  Critical  Design  Reviews:  and  acquisition  milestones. 

The  Department' s  new  manufacturing  readiness  criteria  will  form  the  basis  for  assessing 
pertinent  science  and  technology  efforts,  and  acquisition  programs  throughout  the 
acquisition  lifecycle  on  programs  of  all  acquisition  categories  These  criteria  will  be  a 
tool  to  identify'  relevant  manufacturing  risks  which  require  mitigation.  These 
manufacturing  readiness  criteria  are  expected  to  be  tailored  for  programs  and  will  be 
included  m  the  Department' s  criteria  for  systems  engineering  technical  reviews:  the 
Department' s  templates  for  Preliminary  Design  Review  Critical  Design  Review  repons: 
and  acquisinon  phase  exit  cittern  These  manufacturing  readiness  criteria  will  also  be 
assessed  as  pan  of  the  Program  Support  Reviews  which  the  Department  conducts  on 
Major  Defense  Acquisition  Programs.  These  reviews  ev  aluate  manufacturing  as  part  of 
an  overall  integrated  program  assessment.  These  manufacturing  readiness  criteria  and 
products  will  be  made  available  to  government  and  industry.  Their  use  by  the  Services 
on  lower  AC'AT  programs  will  also  be  encouraged  The  Navy's  Gate  Review  process 
currently  assesses  manufacturing  nsks  but  is  being  updated  with  the  new  manufacturing 
readiness  enteria. 
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DoD  Response  to  GAO-10-439  Recommendations 

RECOMMENDATION*  2:  Tbe  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Office  of  the  Director.  Defense  Research  and  Engineering  to  examine  strengthening 
the  MRL  criteria  related  to  the  process  capability  and  control  of  critical  components 
and  or  interfaces  pnor  to  the  Milestone  C  low  rate  initial  production  decision. 

DOD  RESPONSE;  Concur  Department  of  Defense  Instruction  5000.02  directs  that 
programs  at  Milestone  C  have  no  significant  manufacturing  risks:  that  manufacturing 
processes  have  been  effectively  demonstrated  in  a  pilot  line  environment:  and 
manufacturing  processes  are  under  control  (if  Milestone  C  is  full-rate  production)  While 
the  Department  notes  that  all  manufacturing  processes  do  not  warrant  the  same  level  of 
process  capability  and  control  appropriate  levels  of  control  are  certainly  warranted  on  a 
case  by  case  basis 

The  Department  will  examine  strengthening  the  manufacturing  readiness  criteria  related 
to  process  capability  and  control  of  critical  components  and  or  interfaces  prior  to  the 
Milestone  C  low  rate  initial  production  decision.  However,  program  offices  and 
contractors  should  continue  to  have  the  latitude  to  jointly'  agree  on  the  targets  and  specific 
process  control  demonstrations  required  on  the  pilot  production  line  during  the 
Engmeenng  and  Manufacturing  Development  to  ensure  success. 

REC  OMMENDATION  3 :  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Office  of  the  Director.  Defense  Research  and  Engmeenng  to  assess  the  need  for 
analytical  models  and  tools  to  support  MRL  assessments 

DOD  RESPONSE:  Concur  The  Department  will  collaborate  with  government  services, 
contractors,  and  academia  to  capture  knowledge  and  provide  improved  tools  for 
government  and  contractor  usage  in  conducting  assessments  of  manufacturing  readiness 
as  part  of  systems  engmeenng  technical  review  s  and  milestone  reviews 

REC  OMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense  assess 
the  adequacy  of  the  manufacturing  workforce  knowledge  and  skills  base  across  the 
military  services  and  defeme  agencies  and  develop  a  plan  to  address  current  and  future 
workforce  gaps 

COD  RESPONSE:  Concur  We  agree  that  the  production.  quality  and  manufacturing 
(PQM)  career  field  has  suffered  erosion,  as  have  other  DoD  career  fields.  The  USD 
(AT&L)  Director  of  Human  Capital  has  launched  a  review  of  the  PQM  career  field 
design  to  identify  the  skills,  knowledge  and  training  required  at  each  level  of  career 
progression  in  order  to  develop  training  courses  and  evaluate  progression  of  anticipated 
DoD  planned  new  hues.  The  Department  has  started  to  implement  hiring  and  retention 
strategies  to  mmgate  die  potential  loss  in  experienced,  senior-level  PQM  talent  and 
increase  the  size  of  the  manufacturing  workforce.  As  pan  of  the  Secretary's  growth 
strategy  and  other  initiatives,  the  PQM  career  field  is  projected  to  grow  approximately 
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APPENDIX  H.  CFR,  TITLE  48,  46.202.4 


Code  of  Federal  Regulations 
FAR  Part  46 

Code  of  Federal  Regulations,  title  48,  46.202.4  is  the  Federal  Acquisition  Regulations 
(FAR)  for  Higher-level  contract  quality  requirements  46.202-4. 

eCFR  Electronic  Code  of  Federal  Regulations,  2015. 

48CFR46.202.4(2015) 

Date  accessed  8/2/15 

e-CFR  data  is  current  as  of  August  28,  2015 

URLs: 

https://www.acquisition.gov/?q=browsefar 

http://www.ecfr.gov/cgi-bin/text- 

idx?SID=b3db0004232a54dfbl41afc9ee3a86af&mc=true&node=se48.1.46  1202  64&rg 
n=div8 


Host: 

http://www.gpo.gov/ 


Code  of  Federal  Regulations  (CFR)  annual  edition  is  the  codification  of  the  general  and 
permanent  rules  published  in  the  Federal  Register  by  the  departments  and  agencies  of  the 
Federal  Government  produced  by  the  Office  of  the  Federal  Register  (OFR)  and  the 
Government  Publishing  Office. 


135 


Higher-level  quality  standards 


URLs: 

http://www.bizfilings.com/tooUdt/sbg/run-a-business/govt-contracts/govt-contractors- 

quality-assurance-standards.aspx 

host: 

www.bizfilings.com 


Higher-level  quality  standards  -  All  government  quality  assurance  requirements  are 
spelled  out  in  Part  46  of  the  Federal  Acquisition  Regulation  (FAR). 

Any  language  that  you  will  see  in  a  bid  or  contract  related  to  quality  control  consists  of 
clauses  extracted  from  this  Part. 

When  a  contract  is  for  complex  or  critical  items,  higher-level  requirements  are  applicable. 

The  contracting  officer  is  responsible  for  identifying  the  higher-level  standard(s)  that  will 
satisfy  the  government’s  requirement. 

Title  48  — *■  Chapter  1  — ►  Subchapter  G  — >  Part  46  — >  Subpart  46.2  — ►  §46.202-4 
Higher-level  quality  standards  46.202-4Higher-level  contract  quality  requirements. 

(a)  Agencies  shall  establish  procedures  for  determining  when  higher-level  contract 
quality  requirements  are  necessary,  for  determining  the  risk  (both  the  likelihood  and  the 
impact)  of  nonconformance,  and  for  advising  the  contracting  officer  about  which  higher- 
level  standards  should  be  applied  and  included  in  the  solicitation  and  contract.  Requiring 
compliance  with  higher-level  quality  standards  is  necessary  in  solicitations  and  contracts 
for  complex  or  critical  items  (see  46.203)  or  when  the  technical  requirements  of  the 
contract  require — 

(1)  Control  of  such  things  as  design,  work  operations,  in-process  controls,  testing,  and 
inspection;  or 

(2)  Attention  to  such  factors  as  organization,  planning,  work  instructions,  documentation 
control,  and  advanced  metrology. 

(b)  Examples  of  higher-level  quality  standards  include  overarching  quality  management 
system  standards  such  as  ISO  9001,  ASQ/ANSI  E4,  ASME  NQA-1,  SAE  AS9100,  SAE 
AS9003,  and  ISO/TS  16949,  and  product  or  process  specific  quality  standards  such  as 
SAE  AS5553. 
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[79  FR  70347,  Nov.  25,  2014,  as  amended  at  80  FR  4994,  Jan.  29,  2015] 

Quality  Requirements  May  Apply  for  Subcontractors 

(Applicable  to  complex  or  critical  items;  contracts  may  be  for  less  than  $100,000). 

You  may  be  thinking  to  yourself,  “If  I  am  just  a  subcontractor,  I  won’t  have  to  do  all  this 
quality  stuff,  will  I?”  Guess  again.  In  many  instances,  a  prime  contractor  will  find  it 
necessary  or  desirable  to  pass  along  the  quality  requirements  to  the  subcontractor.  Why? 
The  prime  contractor  is  responsible  for  the  quality  of  materials  supplied  by  the 
subcontractors  or  suppliers,  and  it  is  in  its  best  interest  to  assure  that  all  suppliers  are 
capable  of  providing  the  materials  and  meeting  the  quality  requirements  of  the  prime 
contract. 

The  only  way  that  the  prime  can  assure  itself  that  you  can  do  quality  work,  on  time  and 
within  budget,  is  to  inspect  your  systems  and  get  them  approved.  The  day  of  the  “pal”  or 
“buddy”  at  the  prime  level  that  will  issue  a  contract  just  on  an  owner’s  assurance  that  the 
company  can  deliver  the  required  product  is  becoming  a  thing  of  the  past.  Many  a  small 
business  that  had  this  type  of  relationship  has  found,  to  their  woe,  that  it  must  still  have 
some  kind  of  quality  control  system  in  place.  So  you  must  market  your  company  in  ways 
that  you  might  not  have  had  to  before. 

To  the  surprise  of  many  contractors  and  subcontractors,  government  contract  quality 
assurance  at  the  subcontractor  level  does  not  relieve  the  prime  contractor  of  any 
responsibilities  under  the  contract  nor  does  it  establish  a  contractual  relationship  between 
the  government  and  the  subcontractor.  So,  if  you  think  that  you  are  getting  out  of  some  of 
the  quality  “stuff’  by  being  a  sub,  think  again. 

The  prime  might,  under  a  special  exception  or  for  a  particular  job,  let  you  slide  by 
without  a  QA  program  for  a  while,  but  it  will  eventually  want  to  see  a  formal  program  in 
place  or  it  won’t  want  to  work  with  you.  Therefore,  you  may  as  well  start  creating  your 
own  program  now,  and  do  it  to  your  satisfaction,  without  having  the  pressure  of  having  to 
create  one  on  the  eve  of  a  bid  contract  that  you  really  want. 


137 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


138 


APPENDIX  I.  FDA  PROCEDURES,  PREMARKET  APPROVAL 


This  item  represents  specific  FDA  regulatory  guidance  documents  for  selected 
medical  devices  and  their  respective  production  part  approval  process — 
REGULATIONS:  ISO  9000,  FAR  PART  15. 


Guidance  Documents  (Medical  Devices  and  Radiation-Emitting  Products)  >  Real-Time  P...  Page  1  of  3 


Homo  Medical  Devices  Device  Advice:  Comprehensive  Regulatory  Assistance  Guidar 


s  (Medical  Pevi 


Medical  Devices 

Real-Time  Premarket  Approval  Application  (PMA)  Supplements 


cm 


PDF  Printer 


Document  issued  on:  April  28,  2006 

This  document  supersedes  and  replaces,  the  "Real-Time"  Review  Program  for  Premarket  Approval  Application  (PMA)  Supplements  document  last 
updated  April  22,  1997  and  Section  II  C,  "PMA  Supplements  Definitions  —  Real  Time  Supplements"  of  the  February  2003  guidance  entitled, 
"Assessing  User  Fees:  PMA  Supplement  Definitions,  Modular  PMA  Fees,  BLA  and  Efficacy  Supplement  Definitions,  Bundling  Multiple  Devices  in  a 
Single  Application,  and  Fees  for  Combination  Products;  Guidance  for  Industry  and  FDA" 

For  questions  regarding  this  document,  contact  Thinh  Nguyen  (CDRH)  at  240-276-4010  or  by  e-mail  at  Thinh.Nguyen@fda.hhs.gov.  For  questions  regarding  the 
application  of  this  guidance  to  devices  regulated  by  the  Center  for  Biologies  Evaluation  and  Research  (CBER),  contact  Leonard  Wilson  at  301-827-0373  or  by  email  at 
leonard.wllson@fda.hhs.gov. 

U.S.  Department  of  Health  and  Human  Services 
Food  and  Drug  Administration 
Center  for  Devices  and  Radiological  Health 
Center  for  Biologic  Evaluation  and  Research 
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Contains  Nonbinding  Recommendations 

Public  Comment 

Written  comments  and  suggestions  may  be  submitted  at  any  time  for  Agency  consideration  to  the  Division  of  Dockets  Management,  Food  and  Drug  Administration, 
5630  Fishers  Lane,  Room  1061,  (HFA-305),  Rockville,  MD,  20852.  When  submitting  comments,  please  refer  to  the  exact  title  of  this  guidance  document.  Comments 
may  not  be  acted  upon  by  the  Agency  until  the  document  is  next  revised  or  updated. 

Additional  Copies 


Table  of  Contents 

Purpose 
Background 

Which  modifications  are  appropriate  for  a  Real-Time  PMA  Supplement? 

How  do  I  request  a  Real-Time  review? 

Is  there  a  user  fee  for  a  Real-Time  PMA  Supplement? 

How  do  I  submit  a  Real-Time  PMA  Supplement? 

What  should  a  Real-Time  PMA  Supplement  contain? 

What  is  the  format  of  a  Real-Time  review  meeting? 

Attachment  I.  Sample  Real-Time  Review  Request 

Guidance  for  Industry  and  FDA  Staff 
Real-Time  Premarket  Approval  Application  (PMA)  Supplements 

This  guidance  represents  the  Food  and  Drug  Administration's  (FDA's)  current  thinking  on  this  topic.  It  does  not  create  or  confer  any  rights  for  or  on 
any  person  and  does  not  operate  to  bind  FDA  or  the  public.  You  can  use  an  alternative  approach  if  the  approach  satisfies  the  requirements  of  the 
applicable  statutes  and  regulations.  If  you  want  to  discuss  an  alternative  approach,  contact  the  FDA  staff  responsible  for  implementing  this 
guidance.  If  you  cannot  identify  the  appropriate  FDA  staff,  call  the  appropriate  number  listed  on  the  title  page  of  this  guidance. 

I.  Purpose 

This  guidance  provides  information  about  the  real-time  review  process  for  premarket  approval  application  (PMA)  supplements  and  outlines  the  procedures  for 
requesting  and  submitting  these  types  of  documents.  This  guidance  supersedes  the  document  entitled  “Real-Time"  Review  Program  for  Premarket  Approval 
Application  (PMA)  Supplements,  Issued  April  22,  1997  (the  1997  document)  and  Section  II  C,  "PMA  Supplements  Definitions  —  Real  Time  Supplements"  of  the 
February  2003  guidance  entitled,  "Assessing  User  Fees:  PMA  Supplement  Definitions,  Modular  PMA  Fees,  BLA  and  Efficacy  Supplement  Definitions,  Bundling  Multiple 
Devices  in  a  Single  Application,  and  Fees  for  Combination  Products;  Guidance  for  Industry  and  FDA." 

FDA's  guidance  documents,  including  this  guidance,  do  not  establish  legally  enforceable  responsibilities.  Instead,  guidances  describe  the  Agency’s  current  thinking  on 
a  topic  and  should  be  viewed  only  as  recommendations,  unless  specific  regulatory  or  statutory  requirements  are  cited.  The  use  of  the  word  "should"  in  Agency 
guidances  means  that  something  is  suggested  or  recommended,  but  not  required. 

II.  Background 

On  February  4, 
provide  an  oppo 

(MDUFMA).2  Subsequently,  FDA  issued  a  guidance  entitled  "Assessing  User  Fees:  PMA  Supplement  Definitions,  Modular  PMA  Fees,  BLA  and  Efficacy  Supplement 
Definitions,  Bundling  Multiple  Devices  in  a  Single  Application,  and  Fees  for  Combination  Products"3  and  invited  comments  on  its  topics,  which  included  Real-Time  PMA 
Supplements.  FDA  received  no  comments  on  Real-Time  PMA  Supplements  in  either  docket. 

In  revising  the  1997  document,  FDA  considered  the  comments  submitted  on  It  and  other  comments  about  Real-Time  review  presented  at  the  MDUFMA  Annual 
Stakeholder  Meetings.4  Our  revision  incorporates  the  criteria  for  Real-Time  PMA  Supplements  set  forth  in  section  737(4)(D)  of  the  Federal  Food,  Drug,  and  Cosmetic 
Act  (the  act)  and  clarifies  the  kinds  of  device  modifications  we  believe  are  appropriate  for  real-time  review.  We  continue  to  invite  comments  on  this  guidance. 

The  Least  Burdensome  Approach 

We  believe  we  should  consider  the  least  burdensome  approach  in  all  areas  of  medical  device  regulation.  This  guidance  reflects  our  careful  review  of  the  relevant 
scientific  and  legal  requirements  and  what  we  believe  is  the  least  burdensome  way  for  you  to  comply  with  those  requirements.  However,  if  you  believe  that  an 
alternative  approach  would  be  less  burdensome,  please  contact  us  so  we  can  consider  your  point  of  view.  You  may  send  your  written  comments  to  the  contact  person 

including  ways  to  contact  him,  can 


http:  \\~\v\v. fdn.gov  MedicalDevices  I  )evicekegulationnnd(  iuidance  GuidanceDocuments  ...  4  18  20  14 
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APPENDIX  J.  FAA  PRODUCTION  APPROVAL  PROCESS 


These  selections  represent  specific  regulatory  guidance  from  the  FAA  for 
production  part  approval  certifications  showing  the  PMA  process  flow  chart  and  then  the 
reference  to  FAR  PART  21;  ISO  9000/  AS  9100  and  includes  unique  FAA  requirements. 


APPENDIX  3.  PMA  PROCESS  PLOW  CHART 


Paragraph*  indicated  in  designated  blocks  above-,  refer  to  the  individual  section  within  this  PMA 
document. 
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FAA  -  Aviation  Production  Approval,  FAA.gov  -  FAR  Part  21;  ISO  9000/  AS 
9100,  includes  unique  FAA  requirements. 


Parts  Manufacturer  Approval  (PMA)  —  Production  Approval  Page  1  of  1 


Federal  Aviation 
Administration 


Parts  Manufacturer  Approval  (PMA) 

Production  Approval 


A  PMA  production  approval  allows  you  to  produce  and  sell  FAA  approved  aircraft  articles  that  are  eligible 
for  installation  on  FAA  type  certificated  products. 

A  PMA  is  not  transferable.  A  PMA  is  valid  until  surrendered,  withdrawn  or  terminated.  One  may  sell  PMA 
data  to  another,  but  the  buyer  must  get  a  separate  PMA. 

Withdrawing  Your  PMA 

If  you  no  longer  want  to  produce  and  sell  PMA  articles,  send  a  letter  to  your  local  Manufacturing  Insnection 
District  Office  (MIDO)  (http://www.fiw.qov/abouVciWlc*  ora, field  offices/mido/)  stating  that  you  wish  to  withdraw 
your  PMA.  Then  you  will  turn  in  your  original  PMA  letter  and  supplements  to  the  FAA.  Contact  your  local 
MIDO  about  your  remaining  inventory  of  material  and  articles. 


Page  last  modified:  July  03,  2012  1:15:19  PM  EDT 

This  page  was  published  at:  http://www.faa.gov/aircraft/air_cert/design_approvals/pma/pma_prod/ 


http:/,  'www.faa.gov;  aircraft  /air  cert  design  approvals  pma  pma  prod 


4/17/2014 
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APPENDIX  K.  MOP  RATIONALE  AND  WEIGHTING  FACTORS 


The  following  table  provides  detailed  measure  of  performance  rationale  used  in 
the  analysis  of  alternatives  concerning  various  industry  best  practices  and  then  a 
summary  of  the  weighting  criteria  applied  to  each  of  the  six  identified  key  items. 


Evaluation  Category:  Manufacturing  Knowledge  gaps  and  Associated 
Measures  of  Performance  Category  with  Best  Practice  Rationale 


Item  MOP  . Rationale 


1.  Quality 

Systems 

Objective  MOP 

Best  Practice  Rationale 

1.1.  QMS  -  ISO 
9000/ AS9 100/ 
ISO/TS  16949 

Formal  Quality  Management  System  maintained  by 
producer,  a)  adjudged  to  be  compliant  by  an  auditor 
internal  or  DCMA  or  b)  3rd  party  registration. 
Prescriptive  quality  systems  are  superior  to  general 
guidance  (TS  16949  is  more  prescriptive  than  ISO 

9000) 

QMS,  APQP  & 
PPAP  w/  3rd 

Party  Compliance 

1.2.  APQP 
Manufacturing 
Development 
Guidelines 

Advanced  product  quality  planning:  Consensus 

Standard,  Guidelines  are  superior  to  unique  planning 
processes.  This  is  acknowledged  in  the  aerospace 
standard  AS  9145  that  is  to  supplant  less  disciplined 
manufacturing  development  guidelines. 

1.3.  PPAP 
Followed  -  Pre- 
Production 

Production  part  approval  process:  Consensus  Standard, 
Required  in  Automotive  Industry  and  the  subject  of  a 
pending  aerospace  standard  AS  9145.  Certification  or 
Warrant  Issued  prior  to  production  demonstrating 
production  capability  is  superior  to  production  approval 
based  on  risk  models. 

2.  Requirements 

Objective  MOP 

Best  Practice  Rationale 

Product  & 

Process  w/  PPAP 
Warrant 

2.1.  Common 
Training  - 
infrastructure 

Common  standards  precede  best  practice  training. 
Institutional  training  external  is  preferred  above 
internal  as  variation  is  reduced.  The  absence  of 
standards  reduces  effectiveness 
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Evaluation  Category:  Manufacturing  Knowledge  gaps  and  Associated 
Measures  of  Performance  Category  with  Best  Practice  Rationale 


Item  MOP  . Rationale 


2.2.  Development 
Activities  - 
APQP 

Manufacturing  development  best  practices  requires  a 
standard  disciplined  approach.  If  there  are  prescriptive 
external  standards  they  are  superior  to  heuristic  non¬ 
standard  practices  of  development. 

2.3.  Warrant 
Approach  -  PPAP 

Production  approval  is  warranted  by  a  qualified  agent. 

A  Standard  approach  to  PPAP  is  superior  to  ad-hoc 
processes.  Demonstration  prior  to  production  approval 
is  a  best  practice. 

2.4.  Product 
/Process  - 
Performance  - 
DFMEA/PFMEA 

Product  Design  Failure  Modes  and  Effects  Analysis  to 
the  component  level  are  superior  to  functional  level. 
Process  Failure  Modes  and  Effects  Analysis  to  the 
process  step  with  controls  for  causes  are  superior. 

Causes  need  to  be  actionable  not  being  able  to  ask  why 
any  further.  Process  is  used  to  identify  key 
characteristics  is  a  best  practice. 

2.5.  Measurement 
System 

Compliant 

Measurement  Systems  must  be  calibrated,  repeatable, 
have  high  reliability  and  suitable  to  the  item  tolerance. 

A  rule  of  thumb  is  to  have  error  be  less  than  l/10th  the 
tolerance.  Error  is  used  to  create  error  bands  inside 
design  tolerance  limits. 

2.6.  Process 
Validation 
(Stability  & 
Control  -  SPC) 

Purposefully  and  statistically  define,  manage  and 
interpret  key  process  parameters  using  SPC.  Process 
must  show  that  it  is  firs  stable  and  in  control. 

3.  Design  &  Risk 

Objective  MOP 

Best  Practice  Rationale 

Design 

Definition, 

Maturity 

3.1.  PDR,  CDR 
%  Drawings 
Completed 

Product  Development  practices  assures  technologies 
are  mature,  designs  are  stable.  One  measure  is  percent 
of  drawings  released.  (Example:  85%  at  PDR  and 

100%  at  CDR).  Another  measure  is  having  drawing 
standards  that  use  geometric  tolerances.  Having  high 
skill  reviews  helps  to  remove  ambiguity  and  outside 
reviews  is  a  best  practice. 
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Evaluation  Category:  Manufacturing  Knowledge  gaps  and  Associated 

Measures  of  Performance  Category  with  Best  Practice  Rationale 

Item  MOP  . Rationale . 

3.2.  Drawing 
Controls  CM 

Configuration  control  is  important  to  a  technical  data 
package  defining  the  design  record.  The  same  care 
applied  to  the  manufacturing  process  definition  is  a 
superior  best  practice. 

3.3.  DOEs  - 
Critical 
Characteristics 
Defined 

Once  design  characteristics  are  understood  and  key 
characteristics  identified,  robust  design  use  parameter 
design  experimentation  as  a  best  practices  to 
understand  complex  tolerances,  finds  interactions,  and 
makes  recommendations  to  improve  design  record. 

Process 

Definition  - 
Stability  and 
Control 

3.4.  Not  Stable: 
PpPpk>=  1.33  - 
LRIP 

If  a  new  process  has  not  demonstrated  statistical 
stability  or  homogeneity  for  manufacturing  or 
fabrication  then  determining  a  processes  potential  Ppk 
is  applied  as  a  best  practices. 

3.5.  Stable:  Cpk 
>=  1.67 -FRP 

If  a  more  mature  process  has  demonstrated  statistical 
stability  and  homogeneity  for  a  manufacturing  process 
a  more  rigorous  demonstration  to  a  Cpk  is  a  best 
practice. 

Risks  Mitigated 
&  Issues 

Resolved 

3.6.  Product  & 
Process 

Maturation 

Manufacturing  readiness  level  assessment  Conducted 
or  Manufacturing  Process  Matured  in  Development.  A 
best  practice  is  to  use  a  knowledge-based  approach  as 
compared  to  a  risk-based  approach  showing  evidence 
of  the  maturation.  Using  an  actual  production  line 
running  at  rate  with  parts  that  meet  requirements  is  a 
best  practice. 

3.7. 

Programmatic 
Corrective  Action 
System 

When  inspection,  test  or  field  failure  occurs  then  a 
disciplined  corrective  action  process  begins  to  prevent 
recurrence.  FMEAs  are  updated  and  Non-recurrence  is 
monitored.  A  closed  loop  corrective  action  system  is  a 
best  practice. 

3.8.  Design 
FMEA& 

PFMEA 

After  FMEAS  are  completed  then  using  Risk  Priority 
Numbers  that  guide  improvement  recommendations  to 
achieve  higher  quality  systems  is  a  best  practice. 

4.  Qualification 

Objective  MOP 

Best  Practice  Rationale 
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Evaluation  Category:  Manufacturing  Knowledge  gaps  and  Associated 
Measures  of  Performance  Category  with  Best  Practice  Rationale 


Item  MOP  . Rationale 


4. 1 .  Measurement 
System 

Evaluation 

This  guide  assists  in  the  assessment  of  a  measurement 
system  that  supports  engineering  and  manufacturing 
process. 

4.2.  Product 
Qualification, 
Acceptance  Test 
Procedure  &  First 
Article  Inspection 
and  Test 

Product  functional  requirements  that  demonstrate 
product  performance  and  reliability  are  paid  of 
verification  of  a  compliant  system.  Dimensional 
compliance  that  follows  AS9102  periodically  assessed 
is  a  best  practice. 

Product  & 

4.3.  Process 
Verification  - 
Attributes  & 
Variables  -  SPC 

Following  an  SPC  reference  guide  that  provides  a  wide 
range  of  statistical  methods  for  effective  monitoring 
and  control  of  manufacturing  processes  is  a  best 
practice. 

Process 

Qualification 

Integration 

Supply  Network 

4.4.  Supply 
Network  -  APQP 

Manufacturing  development  best  practices  requires  a 
standard  disciplined  approach.  If  there  are  prescriptive 
external  standards  then  they  are  superior  to  heuristic 
non-standard  practices  for  development  of 
manufacturing.  Supplier  requirements  for  APQP  and 
Control  Plan  guide  are  a  best  practice  that  streamlines 
the  quality  &  manufacturing  process  control  approach 
in  support  of  a  development  program. 

4.5.  Supply 
Network  -  FMEA 

This  guide  assists  in  the  assessment  of  a  design  or  a 
process  that  supports  eliminating  or  reducing  the  effects 
of  failure  modes  identified.  Product  Design  Failure 
Modes  and  Effect  Analysis  to  the  component  level  are 
superior  to  functional  level.  Process  Failure  Modes  and 
Effects  Analysis  to  the  process  step  with  controls  for 
causes  are  superior.  Causes  need  to  be  actionable  not 
being  able  to  ask  why  any  further.  Process  is  used  to 
identify  key  characteristics  is  a  best  practice. 
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Evaluation  Category:  Manufacturing  Knowledge  gaps  and  Associated 
Measures  of  Performance  Category  with  Best  Practice  Rationale 


Item  MOP  . Rationale 


4.6.  Supply 
Network  -  MSA 

This  guide  assists  in  the  assessment  of  a  measurement 
system  that  supports  engineering  and  manufacturing 
process.  Measurement  Systems  must  be  calibrated, 
repeatable,  have  high  reliability  and  suitable  to  the  item 
tolerance.  A  rule  of  thumb  is  to  have  error  be  less  than 
l/10th  the  tolerance.  Error  is  used  to  create  error  bands 
inside  design  tolerance  limits. 

4.7.  Supply 
Network  -  SPC 

This  guide  describes  how  SPC  provides  a  wide  range  of 
statistical  methods  for  effective  monitoring  and  control 
of  manufacturing  processes.  Purposefully  and 
statistically  define,  manage  and  interpret  key  process 
parameters  using  SPC.  Process  must  show  that  it  is  firs 
stable  and  in  control. 

4.8.  Supply 
Network  -  PPAP 

The  supply  network  is  required  to  comply  with  the 
requirements  of  the  PPAP.  Consistent  quality  is 
demonstrated  in  an  actual  production  run  at  production 
rates.  The  Production  part  approval  process  integrates 
production  readiness  including  the  design, 
qualification,  process  capability  with  a  certification 
warrant. 

5.  Metrics 

Objective  MOP 

Best  Practice  Rationale 

Corrective  Action 
System 

Non- 

Conformance  / 
Yield  /Scrap 

5.1.  Field 

Failures  - 
FRACAS 

Field  failures  in  development,  qualification,  field  test 
should  be  following  a  failure  reporting  and  corrective 
action  system.  Problems  could  be  design  or 
manufacturing.  Having  cross  functional  teams  review 
failure  items  should  help  identify  root  cause  and 
mitigate  the  problems  as  a  best  practice.  Reliability 
improvement  and  fix  effectiveness  are  expected 
outcomes. 

5.2.  Process 
Failures  -  Non- 
Conformance 
FRACAS 

Same  as  for  field  failures,  but  process  events. 
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Item  MOP  . Rationale 


5.3.  Yield  /  Scrap 
Metrics  -  Targets 
-  Achievements 

Processes  should  have  a  full  complement  of  metrics 
that  establish  targets,  measure  and  monitor  results  and 
report  to  management.  Best  Practice  quality  metrics 
help  identify  causes  for  shortcomings  and  have 
executive  leadership  involved  in  the  management  of  the 
improvement  processes. 

5.4.  Lean  /  Six 
Sigma  Results  - 
DIMAC 

Six-sigma  is  a  data  driven  analysis  process  that 
quantifies  problems  and  improvement  change  process. 

A  best  practice  uses  six-sigma  to  eliminate  waste  at  the 
enterprise  level  of  management  as  a  best  practice. 

5.5.  Qualification 
-  Test 

Design  Requirements  have  been  defined  and  the 
configured  baseline  has  been  tested  in  field  usage 
conditions  demonstrating  those  requirements.  A  best 
practice  uses  production  representative  or  actual 
production  items. 

Reliability  / 
Durability 

5.6.  Field 
Performance  - 
Warranty 

Post  production  field  usage  data  is  maintained  for  items 
that  fail  in  the  field.  Items  manage  durability  risk 
through  warranty.  The  post  production  data  is  used  to 
confirm  or  improve  reliability  over  time  as  a  best 
practice,  particularly  for  durable  goods. 

5.7.  Things  Gone 
Wrong  (TGW)  / 
Right  (TGR) 

Customer  feedback  is  actively  understood  by  user 
and  internal  data  streams  using  items  like  surveys, 
warranties,  peer  reviews  and  lessons  learned.  There  is 
an  active  activity  to  use  these  feedback  systems  to 
identify  TGW  and  TGR  to  improve  product  and 
process  performance  at  the  Enterprise  and  project  levels 
as  a  best  practice. 

6.  Satisfaction 

Objective  MOP 

Best  Practice  Rationale 
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Item  MOP  . Rationale 


6.1  PAF  Model 
of  Quality 
Improvement 

1 .  Metrics 
Established  - 
Enterprise 

Cost  of  Quality  metrics  is  defined  at  the  enterprise  level 
and  is  a  best  practice.  Quality  metrics  are  financial  and 
product  related  and  are  aligned  to  prevention,  appraisal 
and  failure  activities  (both  internal  and  external).  An 
ABC  cost  accounting  practice  is  aligned  to  PAF  at  the 
local  project  level  is  a  best  practice. 

Quality 

Economics 

6.2  PAF  Model 
of  Quality 
Improvement 

2.  Metrics 
Established  - 
Local 

Cost  of  Quality  metrics  is  defined  at  the  local  level  and 
is  a  best  practice.  Quality  metrics  are  financial  and 
product  related  and  are  aligned  to  prevention,  appraisal 
and  failure  activities  (both  internal  and  external).  An 
ABC  cost  accounting  practice  is  aligned  to  PAF  at  the 
local  project  level  is  a  best  practice. 

6.3  PAF  Model 
of  Quality 
Improvement 

3.  PAF  Managed 

Cost  of  Quality  metrics  including  prevention,  appraisal 
and  failure  activities  (both  internal  and  external)  are 
used  to  influence  product  improvements  and  ROI  by 
activity  in  a  balanced  way  between  PAF. 

Business 

Economics 

6.4.  Sales 
Performance 

Enterprise  level  sales  performance  can  be  traced  to 
product  performance  over  time  and  can  be  related  to 
quality  economics  as  a  best  practice.  ROI  are  related  to 
improvement  actions  and  can  be  readily  determined 
based  on  item  performance  and  from  internal  actions, 
field  activity  or  warranty  systems. 

6.5.  Market  Share 
Performance 

Enterprise  and  local  level  understanding  of  market 
share  is  known  and  related  to  Enterprise  and  project 
performance.  Effects  of  Quality  Economics  can  be 
related  to  market  share  performance  as  a  best  practice. 

149 


II.  Weighting  Factor  Mapping  of  11  Attributes  to  37  Best  Practices 


AO  A  Weigting  factors  determination  based  on  how  many  Best 
Practice  Attributes  can  be  associated  with  an  Evaluation  Category 


Normative  Production  Part  Approval  Process  -  Best  Practice  Attributes 

Input 

A  common  engineering  language. 

i 

A  common  set  of  quality  standards. 

2  1 

A  common  set  of  customer  requirements. 

3 

A  common  3rd  party  compliance  to  quality  system  standards. 

4 

A  common  set  of  prescriptive  advanced  quality  practices. 

5  1 

A  flow  down  of  quality  requirements  to  the  entire  supply  network. 

6 

A  warrant  /  certification  of  demonstrated  capability  prior  to  production. 

7  « 

3 

a. 

A  single  quality  system  assures  administrative  efficiencies. 

8  l 

A  knowledge  based  demonstrated  of  manufacturing  capability. 

9 

A  system  of  quality  metrics  demonstrates  measurable  user  satisfaction. 

10 

A  product's  reliability  and  durability  with  field  performance  tracking. 

11 

Evaluation  Category 

Manufacturing  Knowledge  Gaps  and  Measures  of  Performance 

1.  Quality  Systems 

Objective  Measures  of  Performance 

QMS.  APQP  &  PPAP  w/  3rd  Party  Compliance 

1.  QMS  -  ISO  9000/  AS9100  /  ISO  TS  16949 

2.  APQP  Manufacturing  Development  Guidelines 

3.  PPAP  Followed  -  Pre-Production 

2.  Requirements  Understood 

Objective  Measures  of  Performance 

Product  &  Process  w/  PPAP  Warrant 

1 .  Common  Training  -  infrastructure 

2.  Development  Activities  -  APQP 

3.  Warrant  Approach  -  PPAP 

4.  Product  /Process  -  Performance  -  FMEA 

5.  Measurement  System  Compliant  (Calibration.  Gage  R&R-MSA) 

6.  Process  Validation  (Stability  &  Control  -  SPC) 

3.  Design-Product/Process  &  Risk 

Objective  Measures  of  Performance 

Design  Definition.  Maturity 

1.  PDR.  CDR  °  o  Drawings  Completed 

2.  Drawing  Controls  CM 

3.  DOEs  -  Critical  Characteristics  Defined 

Process  Definition  -  Stability  and  Control 

4.  Not  Stable:  Pp  Ppk  >=  1.33  -  LRIP 

5.  Stable:  Cp  Cpk  >=  1.67  -  FRP 

Risks  Mitigated  &  Issues  Resolved 

6.  Product  &  Process  Maturation 

7.  Programmatic  Corrective  Action  System 

8.  Design  FMEA  &  PFMEA 

4.  Product  &  Process  Qualification 

Objective  Measures  of  Performance 

Product  &  Process  Qualification 

Integration 

Supply  Network 

1.  Measurement  System  Evaluation 

2.  Product  Qualification  -  Qual.  ATP  &  FAIT 

3.  Process  Verification  -  Attributes  &  Variables  -  SPC 

4.  Supply  Network  -  APQP 

5.  Supply  Network  -  MSA 

6.  Supply  Network  -  SPC 

7.  Supply  Network  -  MSA 

8.  Supply  Network  -  PPAP 

5.  Process  &  Product  /  Metrics 

Objective  Measures  of  Performance 

Corrective  Action  System 

Non-Conformance  Yield  /Scrap 

1.  Field  Failures  -  FRACAS 

2.  Process  Failures  —  Non-Conformance  FRACAS 

3.  Yield  /  Scrap  Metrics  -  Targets  -  Achievements 

4.  Lean  /  Six  Sigma  Results  -  DIMAC 

Reliability  /Durability 

5.  Qualification  -  Test 

6.  Field  Performance  -  Warranty 

7.  Things  Gone  Wrong  /  Right 

6.  Customer  Satisfaction 

Objective  Measures  of  Performance 

Quality  Economics 

PAF  Model  of  Quality'  Improvement 

1 .  Metrics  Established  -  Enterprise 

2.  Metrics  Established  -  Local 

3.  PAF  Managed 

Business  Economics 

4.  Sales  Performance 

5.  Market  Share  Performance 
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APPENDIX  L.  LINKS  TO  QUALITY  PRACTICES 


A  standard  Google  search  was  made  to  identify  any  relevant  links  to  quality 
management  systems  that  use  a  Production  Part  Approval  Process  (PPAP).  The  United 
Technologies  Corporation  (UTC)  was  one  organization  identified  as  noted  in  this 
selection. 

I.  UTC  Production-part-approval-process  (UPPAP) 
www.utc.com/Suppliers/Documents/asqr09_2.pdf 

Production-part-approval-process  (PPAP) ...  property  of  United  Technologies 
Corporation  (UTC).  You  may  not  possess,  use,  copy  or  disclose  this  document  or  any 

•  [PDF] 

II.  UPPAP  Basics  issued:  1-13-2016  -  United  . . . 
supplierdiversity.utc.com/Suppliers/Documents/ppap_asqr09_2_faq.pdf 

United  Technologies  Corporation  Farmington,  CT  06032  UPPAP  FAQ  UPPAP  Basics 
issued:  1-13-2016  What  is  UPPAP?  The  UPPAP  process  is  used  to  ensure  that  parts  ... 

III.  •  UPPAP  Toolbox  I  Suppliers  I  United  Technologies 
w  w  w .  utc .  com/S  uppliers/Page  s/UPP  AP-T  oolbox .  aspx 

Download  documents  in  the  UPPAP  Toolbox.  ...  We  are  a  company  founded  by  people 
who  invented  technologies,  turned  them  into  businesses  and,  through  innovation,  built ... 

•  [PDF] 

IV.  TITLE  UTC  Production-part-approval-process  (UPPAP) 
utcaerospacesystems.com/sites/kiddeaerospace/Documents/asqr09_2... 

which  produces  parts  for  a  United  Technologies  Corporation  (UTC)  Member  Company. 

...  Note:  UPPAP  is  modeled  after  AIAG’s  PPAP  (Production-part-approval-process... 

•  [PDF] 

V.  SUPPLIER  QUALITY  MANUAL  Exhibit  1 
https://files.ccs.utc.com/ccs/en/worldwide/contentimages/SQM-BIS... 

Lurther  the  SQM  applies  to  internal  suppliers  within  United  Technologies  ...  PPAP 
submission  should  be  made  as  far  in  advance  of  production  start-up  as  possible,  ... 

•  [PDF] 

VI.  Process  Sheets  /  Supplier  ...  -  United  Technologies 
utc. aws-stage.com/Suppliers/Documents/asqr09_l. pdf 

Business  Entities  need  not  be  re-submitted,  for  new  approval,  until  such  time  ... 
Unpublished  Work  -  ©  UNITED  TECHNOLOGIES  CORPORATION  2012 

VII.  •  Home  I  UTC  Aerospace  Systems 
utcaerospacesystems.com/Pages/Default.aspx 

UTC  Aerospace  Systems  provides  innovative  aerospace  technologies  and  integrated 
systems  that  advance  the  performance,  safety  and  efficiency  of  commercial... 
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APPENDIX  M.  MRL  LEVELS  6  TO  10 


This  selection  describing  the  MRL  levels  6  to  10  are  directly  taken  from  the 
OSD’s  MRL  Deskbook,  version  2.3. 

Manufacturing  Readiness  Level  (MRL)  Deskbook 
Version  2.3 
May  2015 

Prepared  by  the  OSD  Manufacturing  Technology  Program 
In  collaboration  with  The  Joint  Service/Industry  MRL  Working  Group 


Materiel  Solution 
Analysis  (MSA)  jL 

Technology  Maturation  and  Risk 

Reduction  (TMRR)  a 

Engineering  &  Mfg  Development 

L  (EMD)  a 

1  Low-Rate  Initial 
production  (LRIP> 

Full-Rate 

Production  (FRP) 

asr  M 

SRR/SFR 

PDR 

CDR 

PRR/SVR 

^  PC*  ^ 

MRL  4 

MRL  5 

MRL  6 

MRL  7 

MRL  8 

MRL  9 

MRL  10 

MRL  6:  Capability  to  produce  a  prototype  [parts]  in  a  production  relevant  environment 

This  MRL  is  associated  with  readiness  for  a  Milestone  B  decision  to  initiate  an  acquisition 
program  by  entering  into  the  Engineering  and  Manufacturing  Development  (EMD)  Phase  of 
acquisition.  Technologies  should  have  matured  to  at  least  TRL  6.  It  is  normally  seen  as  the  level 
of  manufacturing  readiness  that  denotes  acceptance  of  a  preliminary  system  design.  An  initial 
manufacturing  approach  has  been  developed.  The  majority  of  manufacturing  processes  have  been 
defined  and  characterized,  but  there  are  still  significant  engineering  and/or  design  changes  in  the 
system  itself.  However,  preliminary  design  has  been  completed  and  producibility  assessments  and 
trade  studies  of  key  technologies  and  components  are  complete.  Prototype  manufacturing 
processes  and  technologies,  materials,  tooling  and  test  equipment,  as  well  as  personnel  skills  have 
been  demonstrated  on  systems  and/or  subsystems  in  a  production  relevant  environment.  Cost, 
yield  and  rate  analyses  have  been  performed  to  assess  how  prototype  data  compare  to  target 
objectives,  and  the  program  has  in  place  appropriate  risk  reduction  to  achieve  cost  requirements 
or  establish  a  new  baseline.  This  analysis  should  include  design  trades.  Producibility 
considerations  have  shaped  system  development  plans.  The  Industrial  Capabilities  Assessment 
(ICA)  for  Milestone  B  has  been  completed.  Long-lead  and  key  supply  chain  elements  have  been 
identified. 


MRL  7 :  Capability  to  produce  [parts]  in  a  production  representative  environment 

This  level  of  manufacturing  readiness  is  typical  for  the  mid-point  of  the  Engineering  and 
Manufacturing  Development  (EMD)  Phase  leading  to  the  Post-CDR  Assessment.  Technologies 
should  be  on  a  path  to  achieve  TRL  7.  System  detailed  design  activity  is  nearing  completion. 
Material  specifications  have  been  approved  and  materials  are  available  to  meet  the  planned  pilot 
line  build  schedule.  Manufacturing  processes  and  procedures  have  been  demonstrated  in  a 
production  representative  environment.  Detailed  producibility  trade  studies  are  completed  and 
producibility  enhancements  and  risk  assessments  are  underway.  The  cost  model  has  been  updated 
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with  detailed  designs,  rolled  up  to  system  level,  and  tracked  against  allocated  targets.  Unit  cost 
reduction  efforts  have  been  prioritized  and  are  underway.  Yield  and  rate  analyses  have  been 
updated  with  production  representative  data.  The  supply  chain  and  supplier  quality  assurance 
have  been  assessed  and  long-lead  procurement  plans  are  in  place.  Manufacturing  plans  and 
quality  targets  have  been  developed.  Production  tooling  and  test  equipment  design  and 
development  have  been  initiated. 

MRL  8:  Pilot  line  capability  demonstrated;  Ready  to  begin  Low  Rate  Initial  Production 

This  level  is  associated  with  readiness  for  a  Milestone  C  decision,  and  entry  into  Low  Rate  Initial 
Production  (LRIP).  Technologies  should  have  matured  to  at  least  TRL  7  or  8.  Detailed  system 
design  is  complete  and  sufficiently  stable  to  enter  low  rate  production.  All  materials,  manpower, 
tooling,  test  equipment  and  facilities  arc  proven  on  pilot  line  and  are  available  to  meet  the 
planned  low  rate  production  schedule.  Manufacturing  and  quality  processes  and  procedures  have 
been  proven  in  a  pilot  line  environment  and  arc  under  control  and  ready  for  low  rate  production. 
Known  producibility  risks  pose  no  significant  challenges  for  low  rate  production.  Cost  model  and 
yield  and  rate  analyses  have  been  updated  with  pilot  line  results.  Supplier  qualification  testing 
and  first  article  inspection  have  been  completed.  The  Industrial  Capabilities  Assessment  for 
Milestone  C  has  been  completed  and  shows  that  the  supply  chain  is  established  to  support  LRIP. 


MRL  9:  Low  rate  production  demonstrated;  Capability  in  place  to  begin  Full  Rate  Production 

At  this  level,  the  system,  component  or  item  has  been  previously  produced,  is  in  production,  or 
has  successfully  achieved  low  rate  initial  production.  Technologies  should  have  matured  to  TRL 
8  or  9.  This  level  of  readiness  is  normally  associated  with  readiness  for  entry  into  Full  Rate 
Production  (FRP).  All  systems  engineering/design  requirements  should  have  been  met  such  that 
there  are  minimal  system  changes.  Major  system  design  features  are  stable  and  have  been  proven 
in  test  and  evaluation.  Materials,  parts,  manpower,  tooling,  test  equipment  and  facilities  are 
available  to  meet  planned  rate  production  schedules.  Manufacturing  process  capability  in  a  low 
rate  production  environment  is  at  an  appropriate  quality  level  to  meet  design  key  characteristic 
tolerances.  Production  risk  monitoring  is  ongoing.  LRIP  cost  targets  have  been  met,  and  learning 
curves  have  been  analyzed  with  actual  data.  The  cost  model  has  been  developed  for  FRP 
environment  and  reflects  the  impact  of  continuous  improvement 

MRL  10:  Full  Rate  Production  demonstrated  and  lean  production  practices  in  place. 

This  is  the  highest  level  of  production  readiness.  Technologies  should  have  matured  to  TRL  9. 
This  level  of  manufacturing  is  normally  associated  with  the  Production  or  Sustainment  phases  of 
the  acquisition  life  cycle.  Engineering/design  changes  are  few  and  generally  limited  to  quality  and 
cost  improvements.  System,  components  or  items  arc  in  full  rate  production  and  meet  all 
engineering,  performance,  quality  and  reliability  requirements.  Manufacturing  process  capability 
is  at  the  appropriate  quality  level.  All  materials,  tooling,  inspection  and  test  equipment,  facilities 
and  manpower  arc  in  place  and  have  met  full  rate  production  requirements.  Rate  production  unit 
costs  meet  goals,  and  funding  is  sufficient  for  production  at  required  rates.  Lean  practices  are  well 
established  and  continuous  process  improvements  are  ongoing. 
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APPENDIX  N.  UNDERSTANDING  THE  AUTOMOTIVE  QMS 


A  collection  of  personal  notes  by  William  Ireland  concerning  the  history  of  the 
QS-9000  Quality  Management  System  from  personal  experience  and  prior  publications 
and  presentations  that  are  given  under  Ireland  in  the  List  of  References  of  this  thesis. 
Notes  compiled  on  5/29/2017: 


The  QS-9000  automotive  QMS  transferred  into  the  ISO/TS- 16949  under  SAE 
management  and  are  used  interchangeably  for  this  research.  These  standards  focused  on 
the  identification  and  application  of  customer  specific  requirements.  In  the  global 
automotive  industry  the  ISO/TS- 16949:2009  through:  2016  continues  a  compatibility 
with  the  specifications  of  Ford,  General  Motors  (GM),  Chrysler  domestically  and 
internationally  the  German  automaker’s  VDA6.1  standard,  French  EAQF  standard  and 
the  Italian  automotive  standard  AVSQ.  The  ISO/TS- 16949  standard  is  the  automotive 
industry’s  uniform  consensus  QMS  (AIAG  2006). 

The  motivation  and  implementation  of  the  automotive  PPAP  provided  context  to 
assist  in  the  identification  of  relevant  similarities  between  the  alternative  non-DOD 
industrial  sectors  and  the  DOD.  Historically,  the  American  automotive  sector  faced  a  dual 
challenge  in  the  1970s  and  1980s.  First,  poor  vehicle  quality  and  reliability  resulted  in  a 
rapid  decline  of  market  share.  The  poor  quality  was  associated  with  the  lack  of  attention 
OEMs  gave  to  quality  in  design  and  manufacturing.  In  addition,  the  industry  was 
ignoring  methods  employed  by  the  Japanese  auto  manufacturers  who  were  adhering  to 
statistical  quality  methods  in  production. 

Actions  to  improve  production  outcomes  in  the  automotive  industry  came  from 
detailed  prescriptive  requirements  and  standardization  of  their  QMS.  The  automotive 
standards  required  suppliers  to  follow  prescriptive  quality  practices  to  satisfy  two  key 
elements:  APQP,  representing  the  development  phase  of  a  manufacturing  process  and  2) 
PPAP,  as  validating  manufacturing  process  requirements.  The  GAO  did  characterize  non- 
DOD  organizations  as  more  successful  when  they  followed  a  more  disciplined 
manufacturing  knowledge-based  development  and  production-approval  approach.  This 
contrasted  to  the  DOD  production  approval  process  that  was  non-standard  and  risk-based. 
The  deficiencies  related  to  DOD  acquisition  practices  represented  a  knowledge  gap  in 
process  capability  failing  to  be  objectively  determined  as  part  of  production  readiness  by 
a  demonstration  prior  to  production  start. 

Insight  into  the  enhancement  and  unification  in  automotive  quality  standards  was 
a  trend  moving  key  technologies  out  of  the  OEMs  to  create  a  burgeoning  supply  base 
where  those  entities  became  global  suppliers.  Examples  are  companies  such  as  Visteon, 
Bendix  Allied  Signal,  Delphi  and  TRW.  Early  attempts  by  each  OEM  to  manage  this 
emerging  supply  network  resulted  in  the  creation  of  individual  OEM  QMS  requirements. 

A  confirming  report  given  by  Timothy  J.  Sturgeon  and  Richard  K.  Lester,  in  their 
February  2003  paper  to  the  World  Bank  Project  on  East  Asia’s  Economic  Future  stated: 
“The  entry  of  GM’s  and  Ford’s  former  component  divisions  into  the  merchant  market  for 
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vehicle  components,  modules,  and  systems  has,  almost  overnight,  created  the  world’s  two 
largest,  most  diversified,  and  geographically  extensive  automotive  suppliers”  (Sturgeon 
and  Lester  2003). 

Here,  the  globalization  strategies  were  enablers  to  overcome  the  challenges  of  an 
Asian  invasion  into  the  American  automotive  sector  supporting  the  divesting  of 
component  technologies  within  the  OEMs  to  the  supply  network.  Just  prior  to  the 
automotive  industry  adoption  of  their  common  set  of  standards,  individual  OEMs  had 
unique  QMS  requirements  for  suppliers.  Suppliers  found  themselves  burdened  with  the 
need  to  comply  with  three  different  QMS  while  essentially  producing  for  a  common 
customer  with  nearly  common  production  technologies  -  -  an  onerous  cost  burden  within 
the  supply  base.  The  auto  industry  realized  a  need  to  improve  their  practices  in  the  supply 
network  and  required  an  adherence  to  a  common  set  of  standards  with  the  hopes  to 
improve  production  outcomes. 

As  the  automotive  OEMs  updated  their  global  QMS  they  also  became  system 
integrators.  This  has  similarly  occurred  in  DOD’s  prime  contractors.  This  meant  that  the 
design  expertise  moved  deeper  into  the  supply  base.  As  suppliers  became  larger  global 
enterprises  the  OEMs  and  prime  government  contractors  pushed  component  know-how 
down  into  the  supply  chain  (Ireland  2000).  What  Ireland  reported  establishes  a  concern 
that  there  is  increased  risk  to  the  OEMs  and  prime  contractors  requiring  a  need  to  assure 
production  readiness  can  be  maintained  at  lower  supply  tiers. 

These  first  efforts  by  the  OEMs  failed  to  provide  the  desired  results  and  led  to  a 
collaborative  effort  in  the  automotive  sector.  With  the  formation  of  the  AIAG  in  1982, 
the  OEMs  consolidated  the  various  QMS  requirements  into  a  unified  set  of  practices  with 
the  goal  to  unburden  suppliers  and  build  a  consistency  in  development  of  product  and 
process.  The  AIAG  delivered  a  unified  body  of  standards  applying  development  and 
support  for  a  common  “production  approval  process.” 

Historically,  it  may  be  important  to  put  the  movement  for  a  unified  QMS  standard 
in  perspective.  The  push  for  standards  in  manufacturing  goes  to  the  war  effort  in  World 
War  II.  The  ISO  formation  occurred  shortly  after,  in  1947.  It  was  not  until  the  mid-1970s 
that  standards  became  more  international.  An  international  view  of  quality  standards  was 
launched  in  1987  with  trading  partners  globally  adopting  the  European  based  common  set 
of  quality  management  standards  known  as  the  ISO-9000  QMS. 

The  state  of  standards  prior  to  this  uniform  approach  included  the  adaptation  of 
national  standards.  The  AIAG  used  the  platform  of  the  ISO-9000  standards  to  forward  the 
automotive  industry  quality  standards  from  a  global  perspective.  While  the  core  of  the 
ISO  approach  was  notionally  accepted  it  did  not  address  the  peculiarities  of  the 
automotive  industry.  The  release  of  the  QS-9000  family  of  standards  was  mandated  with 
its  prescriptive  measures  of  compliance  for  production  demonstration  and  control. 
Insightfully,  the  automotive  standards  approach  preserved  the  belief  that  the  global 
supplier  network  would  have  to  be  ISO-9000  certified  by  the  mid-90s  to  trade  with 
Europe.  Therefore,  the  American  automotive  effort  of  standards  development  used  ISO- 
9001  as  its  core  to  its  QMS  requirement  and  then  added  their  unique  requirements.  This 
would  minimize  any  cost  for  implementation  in  DOD  acquisition  being  able  to  build  on 
the  existing  QMS  framework  that  is  based  on  the  ISO-9000. 
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